2024-03-29T10:59:12.420 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:59:12.420 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_amathuri@teuthology --verbose --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622678 --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622678/orig.config.yaml 2024-03-29T10:59:12.457 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:59:12.549 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622678 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622678' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK8KF8LxGDTr84GrjnvlzOw6qKh2eoa6jz9fKWQDjuMowPMVTuuGN5hLqxNs4JjbYvSy47O9KcB53fNM3ZmiQp8= smithi184.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBChIWo4sRluXRqRfde3HW9mpN2k3+EawSGeT0X7df0O5eilEPv32w+Elg4a1z+ulVm2/6dGkwu0juAUsVblJt10= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:59:12.549 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:59:12.549 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:59:12.549 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:59:12.551 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:59:12.574 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:59:12.574 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:59:12.574 INFO:teuthology.packaging:ref: None 2024-03-29T10:59:12.574 INFO:teuthology.packaging:tag: None 2024-03-29T10:59:12.574 INFO:teuthology.packaging:branch: main 2024-03-29T10:59:12.574 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:59:12.574 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:59:12.854 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:59:12.855 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:59:12.862 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:59:12.862 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:59:12.871 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:59:12.884 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:59:12.890 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:59:12.912 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622678', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:54:10.697203', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK8KF8LxGDTr84GrjnvlzOw6qKh2eoa6jz9fKWQDjuMowPMVTuuGN5hLqxNs4JjbYvSy47O9KcB53fNM3ZmiQp8='} 2024-03-29T10:59:12.935 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622678', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:54:10.695910', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBChIWo4sRluXRqRfde3HW9mpN2k3+EawSGeT0X7df0O5eilEPv32w+Elg4a1z+ulVm2/6dGkwu0juAUsVblJt10='} 2024-03-29T10:59:12.935 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:59:12.941 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:59:12.941 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:59:12.941 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:59:13.017 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe8d18220d0>, signals=[15]) 2024-03-29T10:59:13.017 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:59:13.025 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:59:13.025 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2024-03-29T10:59:13.027 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:59:13.116 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi184.front.sepia.ceph.com 2024-03-29T10:59:13.117 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:59:13.203 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:59:13.210 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2024-03-29T10:59:13.229 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2024-03-29T10:59:13.230 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2024-03-29T10:59:13.285 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream" 2024-03-29T10:59:13.285 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="9" 2024-03-29T10:59:13.285 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2024-03-29T10:59:13.285 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:59:13.285 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="9" 2024-03-29T10:59:13.285 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:59:13.285 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:59:13.286 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2024-03-29T10:59:13.286 INFO:teuthology.orchestra.run.smithi170.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:59:13.286 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:59:13.286 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:59:13.286 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:59:13.286 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:59:13.286 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:59:13.287 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2024-03-29T10:59:13.305 DEBUG:teuthology.orchestra.run.smithi184:> uname -m 2024-03-29T10:59:13.322 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2024-03-29T10:59:13.322 DEBUG:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2024-03-29T10:59:13.379 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="CentOS Stream" 2024-03-29T10:59:13.379 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="9" 2024-03-29T10:59:13.379 INFO:teuthology.orchestra.run.smithi184.stdout:ID="centos" 2024-03-29T10:59:13.379 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:59:13.379 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="9" 2024-03-29T10:59:13.380 INFO:teuthology.orchestra.run.smithi184.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:59:13.380 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:59:13.380 INFO:teuthology.orchestra.run.smithi184.stdout:ANSI_COLOR="0;31" 2024-03-29T10:59:13.380 INFO:teuthology.orchestra.run.smithi184.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:59:13.380 INFO:teuthology.orchestra.run.smithi184.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:59:13.380 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:59:13.380 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:59:13.380 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:59:13.380 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:59:13.380 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2024-03-29T10:59:13.405 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:59:13.414 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:59:13.424 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:59:13.424 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:59:13.426 DEBUG:teuthology.orchestra.run.smithi184:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:59:13.443 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:59:13.449 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:59:13.449 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:59:13.480 DEBUG:teuthology.orchestra.run.smithi184:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:59:13.568 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:59:13.615 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:59:13.630 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:59:13.630 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:59:13.631 DEBUG:teuthology.orchestra.run.smithi170:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:59:13.631 DEBUG:teuthology.orchestra.run.smithi184:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:59:13.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:59:13.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:59:13.648 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2024-03-29T10:59:13.649 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2024-03-29T10:59:13.704 INFO:teuthology.orchestra.run.smithi184.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:59:13.705 INFO:teuthology.task.kernel:Running kernel on smithi184: 5.14.0-361.el9.x86_64 2024-03-29T10:59:13.705 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum install -y kernel 2024-03-29T10:59:13.706 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:59:13.706 INFO:teuthology.task.kernel:Running kernel on smithi170: 5.14.0-361.el9.x86_64 2024-03-29T10:59:13.706 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum install -y kernel 2024-03-29T10:59:15.788 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T10:59:15.814 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T10:59:17.455 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-29T10:59:17.458 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-29T10:59:18.320 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-29T10:59:19.274 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 8.0 MB 00:01 2024-03-29T10:59:22.292 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 19 MB 00:01 2024-03-29T10:59:26.444 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - CRB 9.7 MB/s | 6.1 MB 00:00 2024-03-29T10:59:27.926 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 16 kB 00:00 2024-03-29T10:59:28.986 INFO:teuthology.orchestra.run.smithi184.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-29T10:59:34.002 INFO:teuthology.orchestra.run.smithi184.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-29T10:59:35.078 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-29T10:59:36.251 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:59:36.251 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:59:36.339 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T10:59:36.348 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T10:59:36.348 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-03-29T10:59:36.349 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T10:59:36.349 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-03-29T10:59:36.349 INFO:teuthology.orchestra.run.smithi184.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:59:36.349 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-03-29T10:59:36.349 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:59:36.349 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:59:36.349 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:59:36.349 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T10:59:36.349 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-03-29T10:59:36.349 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T10:59:36.349 INFO:teuthology.orchestra.run.smithi184.stdout:Install 4 Packages 2024-03-29T10:59:36.349 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T10:59:36.350 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 76 M 2024-03-29T10:59:36.350 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 123 M 2024-03-29T10:59:36.350 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-03-29T10:59:36.495 INFO:teuthology.orchestra.run.smithi184.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T10:59:36.987 INFO:teuthology.orchestra.run.smithi184.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 24 MB/s | 15 MB 00:00 2024-03-29T10:59:37.537 INFO:teuthology.orchestra.run.smithi184.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 28 MB/s | 34 MB 00:01 2024-03-29T10:59:37.904 INFO:teuthology.orchestra.run.smithi184.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 20 MB/s | 28 MB 00:01 2024-03-29T10:59:37.908 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:37.908 INFO:teuthology.orchestra.run.smithi184.stdout:Total 49 MB/s | 76 MB 00:01 2024-03-29T10:59:37.908 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-03-29T10:59:38.245 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-03-29T10:59:38.245 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-03-29T10:59:39.072 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-03-29T10:59:39.073 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-03-29T10:59:39.303 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 6.1 MB 00:00 2024-03-29T10:59:40.914 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-29T10:59:41.410 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-03-29T10:59:41.964 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-29T10:59:42.060 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:59:42.146 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:59:44.980 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:59:45.126 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:59:46.914 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-03-29T10:59:49.154 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:59:49.154 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:59:49.244 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T10:59:49.251 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T10:59:49.251 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-29T10:59:49.252 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T10:59:49.252 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-29T10:59:49.252 INFO:teuthology.orchestra.run.smithi170.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:59:49.252 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-29T10:59:49.252 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:59:49.252 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:59:49.252 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:59:49.252 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T10:59:49.252 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-29T10:59:49.252 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T10:59:49.252 INFO:teuthology.orchestra.run.smithi170.stdout:Install 4 Packages 2024-03-29T10:59:49.252 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T10:59:49.253 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 76 M 2024-03-29T10:59:49.253 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 123 M 2024-03-29T10:59:49.253 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-29T10:59:49.523 INFO:teuthology.orchestra.run.smithi170.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 710 kB/s | 192 kB 00:00 2024-03-29T10:59:49.856 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:59:49.891 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:50.224 INFO:teuthology.orchestra.run.smithi170.stdout:(2/4): kernel-modules-core-5.14.0-434.el9.x86_6 40 MB/s | 28 MB 00:00 2024-03-29T10:59:50.416 INFO:teuthology.orchestra.run.smithi170.stdout:(3/4): kernel-core-5.14.0-434.el9.x86_64.rpm 13 MB/s | 15 MB 00:01 2024-03-29T10:59:50.941 INFO:teuthology.orchestra.run.smithi170.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 20 MB/s | 34 MB 00:01 2024-03-29T10:59:50.944 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:50.944 INFO:teuthology.orchestra.run.smithi170.stdout:Total 45 MB/s | 76 MB 00:01 2024-03-29T10:59:50.945 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-29T10:59:51.312 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-29T10:59:51.312 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-29T10:59:52.150 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-29T10:59:52.150 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-29T10:59:54.437 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-29T10:59:54.612 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:55.029 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:59:55.117 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:59:57.878 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:59:58.013 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:00:02.722 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:00:02.853 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:00:07.564 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:00:23.791 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:00:23.791 INFO:teuthology.orchestra.run.smithi184.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:00:23.791 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:00:23.834 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:00:24.272 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:00:24.272 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:00:24.273 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:00:24.273 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:00:24.727 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:00:24.727 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:00:24.727 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-03-29T11:00:24.727 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:00:24.727 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:00:24.727 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:00:24.727 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:00:24.727 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:00:24.727 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:00:25.063 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:00:25.155 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:00:25.156 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:00:25.156 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:00:25.156 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:00:25.156 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:00:25.156 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:00:25.156 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi184.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:00:25.157 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:00:25.237 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:00:25.237 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:00:25.238 DEBUG:teuthology.orchestra.run.smithi184:> sudo rpm -qi grub2-tools 2024-03-29T11:00:25.271 INFO:teuthology.orchestra.run.smithi184.stdout:Name : grub2-tools 2024-03-29T11:00:25.271 INFO:teuthology.orchestra.run.smithi184.stdout:Epoch : 1 2024-03-29T11:00:25.271 INFO:teuthology.orchestra.run.smithi184.stdout:Version : 2.06 2024-03-29T11:00:25.272 INFO:teuthology.orchestra.run.smithi184.stdout:Release : 68.el9 2024-03-29T11:00:25.272 INFO:teuthology.orchestra.run.smithi184.stdout:Architecture: x86_64 2024-03-29T11:00:25.272 INFO:teuthology.orchestra.run.smithi184.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:00:25.272 INFO:teuthology.orchestra.run.smithi184.stdout:Group : Unspecified 2024-03-29T11:00:25.272 INFO:teuthology.orchestra.run.smithi184.stdout:Size : 8264254 2024-03-29T11:00:25.272 INFO:teuthology.orchestra.run.smithi184.stdout:License : GPLv3+ 2024-03-29T11:00:25.272 INFO:teuthology.orchestra.run.smithi184.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:00:25.272 INFO:teuthology.orchestra.run.smithi184.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:00:25.272 INFO:teuthology.orchestra.run.smithi184.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:00:25.272 INFO:teuthology.orchestra.run.smithi184.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:00:25.272 INFO:teuthology.orchestra.run.smithi184.stdout:Packager : builder@centos.org 2024-03-29T11:00:25.272 INFO:teuthology.orchestra.run.smithi184.stdout:Vendor : CentOS 2024-03-29T11:00:25.272 INFO:teuthology.orchestra.run.smithi184.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:00:25.273 INFO:teuthology.orchestra.run.smithi184.stdout:Summary : Support tools for GRUB. 2024-03-29T11:00:25.273 INFO:teuthology.orchestra.run.smithi184.stdout:Description : 2024-03-29T11:00:25.273 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:00:25.273 INFO:teuthology.orchestra.run.smithi184.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:00:25.273 INFO:teuthology.orchestra.run.smithi184.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:00:25.273 INFO:teuthology.orchestra.run.smithi184.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:00:25.273 INFO:teuthology.orchestra.run.smithi184.stdout:hardware devices. 2024-03-29T11:00:25.273 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:00:25.273 INFO:teuthology.orchestra.run.smithi184.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:00:25.274 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:00:25.274 INFO:teuthology.task.kernel:Updating grub on smithi184 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:00:25.275 DEBUG:teuthology.orchestra.run.smithi184:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:00:25.924 INFO:teuthology.orchestra.run.smithi184.stderr:Generating grub configuration file ... 2024-03-29T11:00:27.287 INFO:teuthology.orchestra.run.smithi184.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:00:27.326 INFO:teuthology.orchestra.run.smithi184.stderr:done 2024-03-29T11:00:27.328 DEBUG:teuthology.orchestra.run.smithi184:> mktemp 2024-03-29T11:00:27.343 INFO:teuthology.orchestra.run.smithi184.stdout:/tmp/tmp.ND6sXANE9H 2024-03-29T11:00:27.344 DEBUG:teuthology.orchestra.run.smithi184:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ND6sXANE9H 2024-03-29T11:00:27.412 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0666 /tmp/tmp.ND6sXANE9H 2024-03-29T11:00:27.555 DEBUG:teuthology.orchestra.remote:smithi184:/tmp/tmp.ND6sXANE9H is 6KB 2024-03-29T11:00:27.605 DEBUG:teuthology.orchestra.run.smithi184:> rm -fr /tmp/tmp.ND6sXANE9H 2024-03-29T11:00:27.619 DEBUG:teuthology.orchestra.run.smithi184:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:00:27.684 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:00:27.684 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:00:27.684 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:00:27.684 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:00:27.685 DEBUG:teuthology.orchestra.run.smithi184:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:00:27.824 DEBUG:teuthology.orchestra.run.smithi184:> sudo shutdown -r now 2024-03-29T11:00:36.867 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:00:36.867 INFO:teuthology.orchestra.run.smithi170.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:00:36.867 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:00:36.910 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:00:37.319 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:00:37.320 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:00:37.320 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:00:37.320 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:00:37.739 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:00:37.739 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:00:37.739 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-29T11:00:37.739 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:00:37.739 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:00:37.739 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:00:37.739 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:00:37.739 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:00:37.739 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:00:38.114 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:00:38.210 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:00:38.210 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:00:38.211 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:00:38.211 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:00:38.211 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:00:38.211 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:00:38.211 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi170.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:00:38.211 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:00:38.292 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:00:38.292 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:00:38.293 DEBUG:teuthology.orchestra.run.smithi170:> sudo rpm -qi grub2-tools 2024-03-29T11:00:38.334 INFO:teuthology.orchestra.run.smithi170.stdout:Name : grub2-tools 2024-03-29T11:00:38.335 INFO:teuthology.orchestra.run.smithi170.stdout:Epoch : 1 2024-03-29T11:00:38.335 INFO:teuthology.orchestra.run.smithi170.stdout:Version : 2.06 2024-03-29T11:00:38.335 INFO:teuthology.orchestra.run.smithi170.stdout:Release : 68.el9 2024-03-29T11:00:38.335 INFO:teuthology.orchestra.run.smithi170.stdout:Architecture: x86_64 2024-03-29T11:00:38.335 INFO:teuthology.orchestra.run.smithi170.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:00:38.335 INFO:teuthology.orchestra.run.smithi170.stdout:Group : Unspecified 2024-03-29T11:00:38.335 INFO:teuthology.orchestra.run.smithi170.stdout:Size : 8264254 2024-03-29T11:00:38.335 INFO:teuthology.orchestra.run.smithi170.stdout:License : GPLv3+ 2024-03-29T11:00:38.336 INFO:teuthology.orchestra.run.smithi170.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:00:38.336 INFO:teuthology.orchestra.run.smithi170.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:00:38.336 INFO:teuthology.orchestra.run.smithi170.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:00:38.336 INFO:teuthology.orchestra.run.smithi170.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:00:38.336 INFO:teuthology.orchestra.run.smithi170.stdout:Packager : builder@centos.org 2024-03-29T11:00:38.336 INFO:teuthology.orchestra.run.smithi170.stdout:Vendor : CentOS 2024-03-29T11:00:38.336 INFO:teuthology.orchestra.run.smithi170.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:00:38.336 INFO:teuthology.orchestra.run.smithi170.stdout:Summary : Support tools for GRUB. 2024-03-29T11:00:38.336 INFO:teuthology.orchestra.run.smithi170.stdout:Description : 2024-03-29T11:00:38.336 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:00:38.336 INFO:teuthology.orchestra.run.smithi170.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:00:38.336 INFO:teuthology.orchestra.run.smithi170.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:00:38.336 INFO:teuthology.orchestra.run.smithi170.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:00:38.336 INFO:teuthology.orchestra.run.smithi170.stdout:hardware devices. 2024-03-29T11:00:38.337 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:00:38.337 INFO:teuthology.orchestra.run.smithi170.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:00:38.338 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:00:38.338 INFO:teuthology.task.kernel:Updating grub on smithi170 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:00:38.338 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:00:38.955 INFO:teuthology.orchestra.run.smithi170.stderr:Generating grub configuration file ... 2024-03-29T11:00:40.364 INFO:teuthology.orchestra.run.smithi170.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:00:40.408 INFO:teuthology.orchestra.run.smithi170.stderr:done 2024-03-29T11:00:40.411 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2024-03-29T11:00:40.427 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.gp47Ck1q6x 2024-03-29T11:00:40.427 DEBUG:teuthology.orchestra.run.smithi170:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gp47Ck1q6x 2024-03-29T11:00:40.496 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.gp47Ck1q6x 2024-03-29T11:00:40.648 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.gp47Ck1q6x is 6KB 2024-03-29T11:00:40.703 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.gp47Ck1q6x 2024-03-29T11:00:40.723 DEBUG:teuthology.orchestra.run.smithi170:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:00:40.799 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:00:40.799 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:00:40.799 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:00:40.799 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:00:40.799 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:00:40.953 DEBUG:teuthology.orchestra.run.smithi170:> sudo shutdown -r now 2024-03-29T11:00:57.838 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:00:57.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-03-29T11:00:57.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:01:10.970 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:01:10.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-29T11:01:10.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:01:16.190 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-03-29T11:01:25.199 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-03-29T11:01:25.200 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:01:28.254 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-03-29T11:01:29.502 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2024-03-29T11:01:38.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-29T11:01:38.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:01:40.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-03-29T11:01:40.257 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:01:41.566 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2024-03-29T11:01:53.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-29T11:01:53.579 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:01:58.686 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-03-29T11:02:13.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-03-29T11:02:13.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:02:14.235 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-03-29T11:02:14.596 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-03-29T11:02:14.596 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:02:14.596 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2024-03-29T11:02:14.610 INFO:teuthology.orchestra.run.smithi184.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:02:14.610 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:02:14.610 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:02:14.610 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:02:15.611 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:02:15.611 DEBUG:teuthology.orchestra.run.smithi184:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:02:15.700 INFO:teuthology.orchestra.run.smithi184.stdout:ttyS1 2024-03-29T11:02:15.751 DEBUG:teuthology.parallel:result is None 2024-03-29T11:02:25.415 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-03-29T11:02:25.828 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-29T11:02:25.828 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:02:25.829 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2024-03-29T11:02:25.842 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:02:25.842 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:02:25.842 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:02:25.842 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:02:26.843 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:02:26.843 DEBUG:teuthology.orchestra.run.smithi170:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:02:26.940 INFO:teuthology.orchestra.run.smithi170.stdout:ttyS1 2024-03-29T11:02:26.970 DEBUG:teuthology.parallel:result is None 2024-03-29T11:02:26.971 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:02:26.979 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:02:26.979 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:02:26.981 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:02:27.004 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:02:27.011 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:02:27.018 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:02:27.018 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:02:27.047 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:02:27.094 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:02:27.101 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:02:27.101 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:02:27.120 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:02:27.161 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:02:27.161 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:02:27.175 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:02:27.181 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:02:27.183 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:02:27.192 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:02:27.192 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:02:27.227 DEBUG:teuthology.orchestra.run.smithi184:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:02:27.267 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:02:27.276 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:02:27.276 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:02:27.309 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:02:27.330 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:02:27.423 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:02:27.485 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:02:27.525 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:02:27.569 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:02:27.569 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:02:27.636 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:02:27.680 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:02:27.753 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:02:27.796 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:02:27.839 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-29T11:02:27.839 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:02:27.905 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2024-03-29T11:02:27.908 DEBUG:teuthology.orchestra.run.smithi184:> sudo service rsyslog restart 2024-03-29T11:02:27.992 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:02:27.995 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:02:28.518 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:02:28.526 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:02:28.526 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:02:28.540 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:02:28.550 DEBUG:teuthology.orchestra.run.smithi170:> sudo service auditd rotate 2024-03-29T11:02:28.591 INFO:teuthology.orchestra.run.smithi170.stdout:Rotating logs: 2024-03-29T11:02:28.593 DEBUG:teuthology.orchestra.run.smithi184:> sudo service auditd rotate 2024-03-29T11:02:28.643 INFO:teuthology.orchestra.run.smithi184.stdout:Rotating logs: 2024-03-29T11:02:28.644 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:02:28.644 DEBUG:teuthology.orchestra.run.smithi170:> /usr/sbin/getenforce 2024-03-29T11:02:28.667 INFO:teuthology.orchestra.run.smithi170.stdout:Permissive 2024-03-29T11:02:28.667 DEBUG:teuthology.orchestra.run.smithi184:> /usr/sbin/getenforce 2024-03-29T11:02:28.689 INFO:teuthology.orchestra.run.smithi184.stdout:Permissive 2024-03-29T11:02:28.690 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi170.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi184.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:02:28.690 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:02:28.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:02:28.734 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:02:28.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:02:28.763 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:02:28.763 DEBUG:teuthology.orchestra.run.smithi170:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:02:28.810 DEBUG:teuthology.orchestra.run.smithi184:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:02:28.850 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:02:28.860 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-29T11:02:28.864 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-29T11:02:28.996 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:02:29.015 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:02:29.016 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi170.front.sepia.ceph.com,smithi184.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:06:43.271 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi170.front.sepia.ceph.com'), Remote(name='ubuntu@smithi184.front.sepia.ceph.com')] 2024-03-29T11:06:43.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-29T11:06:43.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:06:43.364 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-03-29T11:06:43.448 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-29T11:06:43.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-03-29T11:06:43.449 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:06:43.533 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-03-29T11:06:43.620 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-03-29T11:06:43.620 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:06:43.631 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:06:43.631 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:06:43.631 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:06:43.634 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:06:43.634 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:06:43.661 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:06:43.678 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:06:43.690 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:06:43.707 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:06:43.707 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: ntpd: command not found 2024-03-29T11:06:43.721 INFO:teuthology.orchestra.run.smithi170.stdout:506 Cannot talk to daemon 2024-03-29T11:06:43.736 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: ntpd: command not found 2024-03-29T11:06:43.737 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:06:43.750 INFO:teuthology.orchestra.run.smithi184.stdout:506 Cannot talk to daemon 2024-03-29T11:06:43.753 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:06:43.766 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:06:43.783 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:06:43.804 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:06:43.807 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:06:43.807 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-03-29T11:06:43.807 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:06:43.807 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:06:43.807 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:06:43.808 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:06:43.834 INFO:teuthology.orchestra.run.smithi184.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:06:43.837 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:06:43.837 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2024-03-29T11:06:43.837 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:06:43.837 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:06:43.837 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:06:43.837 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:06:43.838 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:06:43.846 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:06:43.846 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:06:43.846 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:06:43.846 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:06:43.852 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:06:43.852 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:06:43.852 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:06:43.853 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:06:43.855 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:06:43.855 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:06:43.995 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:06:43.996 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:06:44.073 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:06:44.073 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:06:44.172 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:06:44.172 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:06:44.172 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:06:44.200 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:06:44.200 DEBUG:teuthology.orchestra.run.smithi170:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:06:44.244 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:06:44.244 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-29T11:06:44.244 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:06:44.268 DEBUG:teuthology.orchestra.run.smithi170:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:06:44.271 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:06:44.271 DEBUG:teuthology.orchestra.run.smithi184:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:06:44.340 DEBUG:teuthology.orchestra.run.smithi184:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:06:44.348 DEBUG:teuthology.orchestra.run.smithi170:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:06:44.415 INFO:teuthology.orchestra.run.smithi170.stdout:check_obsoletes = 1 2024-03-29T11:06:44.416 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum clean all 2024-03-29T11:06:44.419 DEBUG:teuthology.orchestra.run.smithi184:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:06:44.486 INFO:teuthology.orchestra.run.smithi184.stdout:check_obsoletes = 1 2024-03-29T11:06:44.488 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum clean all 2024-03-29T11:06:44.662 INFO:teuthology.orchestra.run.smithi170.stdout:74 files removed 2024-03-29T11:06:44.691 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-radosgw 2024-03-29T11:06:44.728 INFO:teuthology.orchestra.run.smithi184.stdout:74 files removed 2024-03-29T11:06:44.754 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-radosgw 2024-03-29T11:06:45.334 INFO:teuthology.orchestra.run.smithi170.stdout:ceph packages for x86_64 240 kB/s | 82 kB 00:00 2024-03-29T11:06:45.359 INFO:teuthology.orchestra.run.smithi184.stdout:ceph packages for x86_64 275 kB/s | 82 kB 00:00 2024-03-29T11:06:45.684 INFO:teuthology.orchestra.run.smithi170.stdout:ceph noarch packages 33 kB/s | 10 kB 00:00 2024-03-29T11:06:45.718 INFO:teuthology.orchestra.run.smithi184.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-29T11:06:46.052 INFO:teuthology.orchestra.run.smithi170.stdout:ceph source packages 5.8 kB/s | 1.9 kB 00:00 2024-03-29T11:06:46.122 INFO:teuthology.orchestra.run.smithi184.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-03-29T11:06:46.512 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T11:06:46.687 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T11:06:47.896 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-29T11:06:48.138 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-29T11:06:48.887 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-29T11:06:50.680 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-03-29T11:06:55.489 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 6.1 MB 00:01 2024-03-29T11:06:56.856 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - Extras packages 88 kB/s | 16 kB 00:00 2024-03-29T11:06:56.894 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 950 kB/s | 8.0 MB 00:08 2024-03-29T11:07:19.368 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 904 kB/s | 19 MB 00:21 2024-03-29T11:07:24.994 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.1 MB 00:02 2024-03-29T11:07:26.903 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-03-29T11:07:28.687 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-29T11:07:33.703 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-29T11:07:36.311 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:07:36.325 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:07:36.325 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-29T11:07:36.325 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:07:36.325 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-29T11:07:36.325 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:07:36.326 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrading: 2024-03-29T11:07:36.326 INFO:teuthology.orchestra.run.smithi170.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:07:36.326 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:07:36.326 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-29T11:07:36.326 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options 2024-03-29T11:07:36.326 INFO:teuthology.orchestra.run.smithi170.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:07:36.326 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:07:36.326 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:07:36.326 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:07:36.327 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:07:36.327 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:07:36.327 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:07:36.327 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:07:36.327 INFO:teuthology.orchestra.run.smithi170.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:07:36.327 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:07:36.327 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:07:36.327 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:07:36.327 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:07:36.328 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:07:36.328 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:07:36.328 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:07:36.328 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:07:36.328 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse 2024-03-29T11:07:36.328 INFO:teuthology.orchestra.run.smithi170.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:07:36.328 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:07:36.328 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:07:36.328 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:07:36.328 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:07:36.328 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:07:36.328 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:07:36.329 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:07:36.329 INFO:teuthology.orchestra.run.smithi170.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:07:36.329 INFO:teuthology.orchestra.run.smithi170.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:07:36.329 INFO:teuthology.orchestra.run.smithi170.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:07:36.329 INFO:teuthology.orchestra.run.smithi170.stdout:Installing weak dependencies: 2024-03-29T11:07:36.329 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:07:36.330 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:07:36.330 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-29T11:07:36.330 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:07:36.330 INFO:teuthology.orchestra.run.smithi170.stdout:Install 30 Packages 2024-03-29T11:07:36.330 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrade 2 Packages 2024-03-29T11:07:36.330 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:07:36.331 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 58 M 2024-03-29T11:07:36.331 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-29T11:07:38.421 INFO:teuthology.orchestra.run.smithi170.stdout:(1/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 10 MB/s | 20 MB 00:01 2024-03-29T11:07:38.521 INFO:teuthology.orchestra.run.smithi170.stdout:(2/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 2.6 MB/s | 5.3 MB 00:02 2024-03-29T11:07:38.537 INFO:teuthology.orchestra.run.smithi170.stdout:(3/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 215 kB/s | 25 kB 00:00 2024-03-29T11:07:38.605 INFO:teuthology.orchestra.run.smithi170.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 7.3 MB/s | 498 kB 00:00 2024-03-29T11:07:38.738 INFO:teuthology.orchestra.run.smithi170.stdout:(5/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 3.3 MB/s | 742 kB 00:00 2024-03-29T11:07:38.839 INFO:teuthology.orchestra.run.smithi170.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 451 kB/s | 45 kB 00:00 2024-03-29T11:07:38.931 INFO:teuthology.orchestra.run.smithi170.stdout:(7/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 16 MB/s | 5.1 MB 00:00 2024-03-29T11:07:38.956 INFO:teuthology.orchestra.run.smithi170.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.1 MB/s | 131 kB 00:00 2024-03-29T11:07:38.990 INFO:teuthology.orchestra.run.smithi170.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.7 MB/s | 162 kB 00:00 2024-03-29T11:07:39.057 INFO:teuthology.orchestra.run.smithi170.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 4.4 MB/s | 298 kB 00:00 2024-03-29T11:07:39.090 INFO:teuthology.orchestra.run.smithi170.stdout:(11/32): python3-rados-19.0.0-1439.g5dcb190d.el 2.3 MB/s | 320 kB 00:00 2024-03-29T11:07:39.115 INFO:teuthology.orchestra.run.smithi170.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 99 kB 00:00 2024-03-29T11:07:39.224 INFO:teuthology.orchestra.run.smithi170.stdout:(13/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 3.5 MB/s | 9.7 MB 00:02 2024-03-29T11:07:39.374 INFO:teuthology.orchestra.run.smithi170.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 373 kB/s | 106 kB 00:00 2024-03-29T11:07:39.400 INFO:teuthology.orchestra.run.smithi170.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 356 kB/s | 44 kB 00:00 2024-03-29T11:07:39.425 INFO:teuthology.orchestra.run.smithi170.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 515 kB/s | 159 kB 00:00 2024-03-29T11:07:39.492 INFO:teuthology.orchestra.run.smithi170.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 661 kB 00:00 2024-03-29T11:07:39.526 INFO:teuthology.orchestra.run.smithi170.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.2 MB/s | 282 kB 00:00 2024-03-29T11:07:39.550 INFO:teuthology.orchestra.run.smithi170.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 328 kB/s | 41 kB 00:00 2024-03-29T11:07:39.576 INFO:teuthology.orchestra.run.smithi170.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 220 kB 00:00 2024-03-29T11:07:39.601 INFO:teuthology.orchestra.run.smithi170.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 929 kB/s | 46 kB 00:00 2024-03-29T11:07:39.627 INFO:teuthology.orchestra.run.smithi170.stdout:(22/32): qatlib-service-24.02.0-1.el9.x86_64.rp 463 kB/s | 35 kB 00:00 2024-03-29T11:07:39.885 INFO:teuthology.orchestra.run.smithi170.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 100 kB/s | 26 kB 00:00 2024-03-29T11:07:40.110 INFO:teuthology.orchestra.run.smithi170.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 216 kB/s | 49 kB 00:00 2024-03-29T11:07:40.136 INFO:teuthology.orchestra.run.smithi170.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 552 kB/s | 309 kB 00:00 2024-03-29T11:07:40.286 INFO:teuthology.orchestra.run.smithi170.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 384 kB/s | 67 kB 00:00 2024-03-29T11:07:40.444 INFO:teuthology.orchestra.run.smithi170.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.7 MB/s | 838 kB 00:00 2024-03-29T11:07:40.511 INFO:teuthology.orchestra.run.smithi170.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 4.4 MB 00:00 2024-03-29T11:07:40.537 INFO:teuthology.orchestra.run.smithi170.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 766 kB/s | 192 kB 00:00 2024-03-29T11:07:40.670 INFO:teuthology.orchestra.run.smithi170.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2024-03-29T11:07:40.870 INFO:teuthology.orchestra.run.smithi170.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 9.4 MB/s | 3.1 MB 00:00 2024-03-29T11:07:41.179 INFO:teuthology.orchestra.run.smithi170.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 5.1 MB/s | 3.4 MB 00:00 2024-03-29T11:07:41.184 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:41.184 INFO:teuthology.orchestra.run.smithi170.stdout:Total 12 MB/s | 58 MB 00:04 2024-03-29T11:07:41.184 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-29T11:07:41.222 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-29T11:07:41.222 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-29T11:07:41.707 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-29T11:07:41.707 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-29T11:07:43.205 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-29T11:07:43.308 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:07:43.335 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:07:43.369 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:07:43.391 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:07:43.408 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:07:43.600 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:07:43.663 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:07:44.014 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:07:44.077 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:07:44.152 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:07:44.241 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:07:44.279 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:07:44.341 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:07:44.370 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:07:44.415 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:07:44.481 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:07:44.506 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:07:44.594 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:07:44.629 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:07:44.737 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:07:44.773 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:07:44.812 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:07:44.846 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:07:44.883 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:07:44.915 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:07:45.223 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:07:45.254 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:07:45.343 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:07:45.375 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:07:45.417 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:07:45.891 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:07:45.924 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:07:45.994 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:07:46.062 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:07:46.092 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:07:48.295 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:07:48.339 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:07:48.880 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:07:48.917 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:07:49.011 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:07:49.011 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:07:49.011 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:07:49.011 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:07:49.051 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:07:58.764 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:07:58.764 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /sys 2024-03-29T11:07:58.765 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /proc 2024-03-29T11:07:58.765 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /mnt 2024-03-29T11:07:58.765 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /var/tmp 2024-03-29T11:07:58.765 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /home 2024-03-29T11:07:58.765 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /root 2024-03-29T11:07:58.765 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /tmp 2024-03-29T11:07:58.765 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:07:59.675 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:07:59.704 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:07:59.704 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:07:59.704 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:07:59.704 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:07:59.704 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:07:59.704 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:07:59.705 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:07:59.778 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:07:59.778 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:08:00.941 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:08:00.941 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:08:00.941 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:08:00.941 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:08:00.941 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:08:00.941 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:08:00.941 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:08:00.942 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:08:00.942 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:08:00.942 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:08:00.942 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:08:00.942 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:08:00.942 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:08:00.942 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:08:00.942 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:08:00.942 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:08:00.942 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:08:00.942 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:08:00.942 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:08:00.942 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:08:00.943 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:08:00.943 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:08:00.943 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:08:00.943 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:08:00.943 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:08:00.943 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:08:00.943 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:08:00.943 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:08:00.943 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:08:00.943 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:08:00.943 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:08:00.944 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:08:00.944 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:08:00.944 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:08:01.992 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:08:01.992 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:01.993 INFO:teuthology.orchestra.run.smithi170.stdout:Upgraded: 2024-03-29T11:08:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:01.993 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-29T11:08:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:08:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:08:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:08:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:08:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:08:01.994 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:08:01.994 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:08:01.994 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:01.994 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:08:01.994 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:01.994 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:08:01.994 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:08:01.994 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:08:01.994 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:01.994 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:01.994 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:01.994 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:08:01.994 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:01.994 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:01.995 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:01.995 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:08:01.995 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:08:01.995 INFO:teuthology.orchestra.run.smithi170.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:08:01.995 INFO:teuthology.orchestra.run.smithi170.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:08:01.995 INFO:teuthology.orchestra.run.smithi170.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:08:01.995 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:01.995 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:08:02.345 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-test 2024-03-29T11:08:02.904 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:08:03.055 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:08:03.056 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:08:03.056 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-29T11:08:03.056 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:08:03.056 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-29T11:08:03.056 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:08:03.056 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-29T11:08:03.056 INFO:teuthology.orchestra.run.smithi170.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:08:03.057 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:08:03.057 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:08:03.057 INFO:teuthology.orchestra.run.smithi170.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:08:03.057 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:08:03.057 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:03.057 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-29T11:08:03.057 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:08:03.057 INFO:teuthology.orchestra.run.smithi170.stdout:Install 6 Packages 2024-03-29T11:08:03.057 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:03.058 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 48 M 2024-03-29T11:08:03.058 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 204 M 2024-03-29T11:08:03.058 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-29T11:08:03.498 INFO:teuthology.orchestra.run.smithi170.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-29T11:08:03.548 INFO:teuthology.orchestra.run.smithi170.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T11:08:03.581 INFO:teuthology.orchestra.run.smithi170.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:08:03.707 INFO:teuthology.orchestra.run.smithi170.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 427 kB/s | 163 kB 00:00 2024-03-29T11:08:03.816 INFO:teuthology.orchestra.run.smithi170.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 802 kB/s | 187 kB 00:00 2024-03-29T11:08:07.124 INFO:teuthology.orchestra.run.smithi170.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 12 MB/s | 47 MB 00:03 2024-03-29T11:08:07.126 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:07.127 INFO:teuthology.orchestra.run.smithi170.stdout:Total 12 MB/s | 48 MB 00:04 2024-03-29T11:08:07.141 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-29T11:08:07.162 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-29T11:08:07.163 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-29T11:08:07.552 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-29T11:08:07.552 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-29T11:08:08.131 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-29T11:08:08.200 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:08:08.238 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:08:08.278 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:08:08.324 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:08:08.352 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:08:12.924 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:08:13.010 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:08:15.574 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:08:15.574 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:08:15.574 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:08:15.574 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:08:15.574 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:08:15.574 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:08:15.931 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:08:15.931 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:15.931 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-29T11:08:15.932 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:15.932 INFO:teuthology.orchestra.run.smithi170.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:08:15.932 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:15.932 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:08:15.932 INFO:teuthology.orchestra.run.smithi170.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:08:15.932 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:08:15.932 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:15.932 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:08:16.239 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph 2024-03-29T11:08:16.812 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:43 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:08:16.967 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:08:16.969 INFO:teuthology.orchestra.run.smithi170.stdout:========================================================================================= 2024-03-29T11:08:16.969 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-29T11:08:16.969 INFO:teuthology.orchestra.run.smithi170.stdout:========================================================================================= 2024-03-29T11:08:16.970 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-29T11:08:16.970 INFO:teuthology.orchestra.run.smithi170.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:08:16.970 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-29T11:08:16.970 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:08:16.970 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:08:16.970 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:08:16.970 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:08:16.970 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:08:16.970 INFO:teuthology.orchestra.run.smithi170.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:08:16.970 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:08:16.970 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:08:16.970 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:08:16.971 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:08:16.971 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:08:16.971 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:08:16.971 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:08:16.971 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:08:16.971 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:08:16.971 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:08:16.971 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:08:16.971 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:08:16.971 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:08:16.971 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:08:16.971 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:08:16.971 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:08:16.972 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:08:16.972 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:08:16.972 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:08:16.972 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:08:16.972 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:08:16.972 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:08:16.972 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:08:16.972 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:08:16.972 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:08:16.972 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:08:16.972 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:08:16.972 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:08:16.972 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:08:16.973 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:16.973 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-29T11:08:16.973 INFO:teuthology.orchestra.run.smithi170.stdout:========================================================================================= 2024-03-29T11:08:16.973 INFO:teuthology.orchestra.run.smithi170.stdout:Install 36 Packages 2024-03-29T11:08:16.973 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:16.973 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 30 M 2024-03-29T11:08:16.973 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 107 M 2024-03-29T11:08:16.974 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-29T11:08:17.821 INFO:teuthology.orchestra.run.smithi170.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 38 kB/s | 6.6 kB 00:00 2024-03-29T11:08:18.621 INFO:teuthology.orchestra.run.smithi170.stdout:(2/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 5.9 MB/s | 4.7 MB 00:00 2024-03-29T11:08:18.671 INFO:teuthology.orchestra.run.smithi170.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.5 MB/s | 1.5 MB 00:01 2024-03-29T11:08:18.822 INFO:teuthology.orchestra.run.smithi170.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.6 MB/s | 247 kB 00:00 2024-03-29T11:08:18.955 INFO:teuthology.orchestra.run.smithi170.stdout:(5/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.6 MB/s | 2.1 MB 00:01 2024-03-29T11:08:19.080 INFO:teuthology.orchestra.run.smithi170.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-03-29T11:08:19.106 INFO:teuthology.orchestra.run.smithi170.stdout:(7/36): lua-5.4.4-4.el9.x86_64.rpm 658 kB/s | 187 kB 00:00 2024-03-29T11:08:19.131 INFO:teuthology.orchestra.run.smithi170.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 632 kB/s | 32 kB 00:00 2024-03-29T11:08:19.156 INFO:teuthology.orchestra.run.smithi170.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 947 kB/s | 47 kB 00:00 2024-03-29T11:08:19.181 INFO:teuthology.orchestra.run.smithi170.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 770 kB/s | 39 kB 00:00 2024-03-29T11:08:19.223 INFO:teuthology.orchestra.run.smithi170.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-29T11:08:19.399 INFO:teuthology.orchestra.run.smithi170.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 5.3 MB/s | 1.1 MB 00:00 2024-03-29T11:08:19.646 INFO:teuthology.orchestra.run.smithi170.stdout:(13/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 16 MB/s | 17 MB 00:01 2024-03-29T11:08:19.666 INFO:teuthology.orchestra.run.smithi170.stdout:(14/36): python3-ply-3.11-14.el9.noarch.rpm 233 kB/s | 103 kB 00:00 2024-03-29T11:08:19.691 INFO:teuthology.orchestra.run.smithi170.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 425 kB/s | 124 kB 00:00 2024-03-29T11:08:19.717 INFO:teuthology.orchestra.run.smithi170.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2024-03-29T11:08:19.834 INFO:teuthology.orchestra.run.smithi170.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 882 kB/s | 125 kB 00:00 2024-03-29T11:08:20.109 INFO:teuthology.orchestra.run.smithi170.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 53 kB/s | 22 kB 00:00 2024-03-29T11:08:20.193 INFO:teuthology.orchestra.run.smithi170.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 121 kB/s | 43 kB 00:00 2024-03-29T11:08:20.326 INFO:teuthology.orchestra.run.smithi170.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 248 kB/s | 151 kB 00:00 2024-03-29T11:08:20.793 INFO:teuthology.orchestra.run.smithi170.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 23 kB/s | 11 kB 00:00 2024-03-29T11:08:20.860 INFO:teuthology.orchestra.run.smithi170.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 537 kB/s | 358 kB 00:00 2024-03-29T11:08:20.877 INFO:teuthology.orchestra.run.smithi170.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 211 kB/s | 18 kB 00:00 2024-03-29T11:08:20.927 INFO:teuthology.orchestra.run.smithi170.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 347 kB/s | 23 kB 00:00 2024-03-29T11:08:20.944 INFO:teuthology.orchestra.run.smithi170.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 290 kB/s | 19 kB 00:00 2024-03-29T11:08:20.994 INFO:teuthology.orchestra.run.smithi170.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 294 kB/s | 20 kB 00:00 2024-03-29T11:08:21.078 INFO:teuthology.orchestra.run.smithi170.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 347 kB/s | 46 kB 00:00 2024-03-29T11:08:21.136 INFO:teuthology.orchestra.run.smithi170.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 556 kB/s | 79 kB 00:00 2024-03-29T11:08:21.220 INFO:teuthology.orchestra.run.smithi170.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 197 kB/s | 16 kB 00:00 2024-03-29T11:08:21.245 INFO:teuthology.orchestra.run.smithi170.stdout:(30/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 151 kB/s | 172 kB 00:01 2024-03-29T11:08:21.362 INFO:teuthology.orchestra.run.smithi170.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 633 kB/s | 90 kB 00:00 2024-03-29T11:08:21.421 INFO:teuthology.orchestra.run.smithi170.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 204 kB/s | 36 kB 00:00 2024-03-29T11:08:21.580 INFO:teuthology.orchestra.run.smithi170.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 543 kB/s | 272 kB 00:00 2024-03-29T11:08:21.621 INFO:teuthology.orchestra.run.smithi170.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 887 kB/s | 230 kB 00:00 2024-03-29T11:08:21.646 INFO:teuthology.orchestra.run.smithi170.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 299 kB/s | 20 kB 00:00 2024-03-29T11:08:22.722 INFO:teuthology.orchestra.run.smithi170.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 328 kB/s | 427 kB 00:01 2024-03-29T11:08:22.725 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:22.726 INFO:teuthology.orchestra.run.smithi170.stdout:Total 5.2 MB/s | 30 MB 00:05 2024-03-29T11:08:22.751 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-29T11:08:22.794 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-29T11:08:22.794 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-29T11:08:23.212 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-29T11:08:23.212 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-29T11:08:24.556 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-29T11:08:24.624 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:08:24.652 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:08:24.684 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:08:24.756 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:08:24.792 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:08:24.825 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:08:24.917 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:08:25.011 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:08:25.052 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:08:25.124 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:08:25.180 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:08:25.225 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:08:25.274 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:08:25.329 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:08:25.513 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:08:25.563 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:08:25.593 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:08:25.628 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:08:25.658 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:08:25.735 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:08:25.768 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:08:25.810 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:08:25.850 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:08:25.885 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:08:25.919 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:08:26.010 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:08:26.055 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:08:26.131 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:08:26.277 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:08:26.403 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:08:26.568 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:08:26.597 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:08:26.623 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:08:26.623 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:08:26.623 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:08:26.623 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:08:26.623 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:08:26.623 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:28.048 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:08:28.075 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:08:28.075 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:08:28.075 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:08:28.075 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:08:28.076 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:08:28.076 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:28.507 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:08:28.531 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:08:28.531 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:08:28.531 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:08:28.531 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:08:28.531 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:08:28.531 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:28.750 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:08:28.773 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:08:28.773 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:08:28.773 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:08:28.773 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:08:28.773 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:08:28.774 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:28.830 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:08:30.632 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:08:30.632 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:08:30.632 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:08:30.632 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:08:30.633 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:08:30.633 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:08:30.633 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:08:30.633 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:08:30.633 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:08:30.633 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:08:30.633 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:08:30.633 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:08:30.633 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:08:30.633 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:08:30.633 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:08:30.633 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:08:30.633 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:08:30.633 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:08:30.633 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:08:30.634 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:08:30.634 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:08:30.634 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:08:30.634 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:08:30.634 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:08:30.634 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:08:30.634 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:08:30.635 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:08:30.635 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:08:30.635 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:08:30.635 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:08:30.636 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:08:30.636 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:08:30.636 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:08:30.636 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:08:30.636 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:08:30.636 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:08:31.647 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:08:31.648 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:31.648 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-29T11:08:31.648 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:31.648 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:31.648 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:31.648 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:08:31.648 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:31.648 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:31.648 INFO:teuthology.orchestra.run.smithi170.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:08:31.648 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:08:31.648 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:08:31.648 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:08:31.648 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:08:31.648 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:08:31.649 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:08:31.649 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:08:31.649 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:08:31.649 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:08:31.649 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:08:31.649 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:08:31.649 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:08:31.649 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:08:31.649 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:08:31.649 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:08:31.649 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:08:31.649 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:08:31.649 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:08:31.649 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:08:31.649 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:08:31.650 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:08:31.650 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:08:31.650 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:08:31.650 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:08:31.650 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:08:31.650 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:08:31.650 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:08:31.650 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:08:31.650 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:08:31.650 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:31.650 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:08:32.016 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-base 2024-03-29T11:08:32.585 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:08:32.685 INFO:teuthology.orchestra.run.smithi170.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:08:32.737 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:08:32.739 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-03-29T11:08:32.739 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:08:32.802 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install cephadm 2024-03-29T11:08:33.360 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:08:33.511 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:08:33.512 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:08:33.512 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-29T11:08:33.512 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:08:33.512 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-29T11:08:33.513 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:08:33.513 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:33.513 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-29T11:08:33.513 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:08:33.513 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-03-29T11:08:33.513 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:33.513 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 747 k 2024-03-29T11:08:33.513 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 754 k 2024-03-29T11:08:33.513 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-29T11:08:33.976 INFO:teuthology.orchestra.run.smithi170.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.6 MB/s | 747 kB 00:00 2024-03-29T11:08:33.977 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:33.977 INFO:teuthology.orchestra.run.smithi170.stdout:Total 1.6 MB/s | 747 kB 00:00 2024-03-29T11:08:33.977 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-29T11:08:33.982 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-29T11:08:33.982 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-29T11:08:33.988 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-29T11:08:33.988 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-29T11:08:34.062 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-29T11:08:34.517 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:08:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:08:34.953 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:08:35.250 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:08:35.251 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:35.251 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-29T11:08:35.251 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:08:35.251 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:35.251 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:08:35.474 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:08:36.043 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:08:36.195 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:08:36.196 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:08:36.196 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2024-03-29T11:08:36.196 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:08:36.196 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-29T11:08:36.196 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:08:36.196 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:36.196 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-29T11:08:36.196 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:08:36.196 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-03-29T11:08:36.196 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:36.196 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 145 k 2024-03-29T11:08:36.196 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 436 k 2024-03-29T11:08:36.197 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-29T11:08:36.433 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 615 kB/s | 145 kB 00:00 2024-03-29T11:08:36.434 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:36.434 INFO:teuthology.orchestra.run.smithi170.stdout:Total 612 kB/s | 145 kB 00:00 2024-03-29T11:08:36.434 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-29T11:08:36.441 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-29T11:08:36.441 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-29T11:08:36.493 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-29T11:08:36.494 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-29T11:08:36.641 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-29T11:08:36.692 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:08:36.716 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:08:36.716 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:08:36.717 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:08:36.717 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:37.690 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:08:37.690 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:37.690 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-29T11:08:37.690 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:37.690 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:37.690 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:08:37.957 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr 2024-03-29T11:08:38.574 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:05 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:08:38.675 INFO:teuthology.orchestra.run.smithi170.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:08:38.727 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:08:38.728 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-03-29T11:08:38.728 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:08:38.792 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:08:39.351 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:08:39.502 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:08:39.503 INFO:teuthology.orchestra.run.smithi170.stdout:====================================================================================== 2024-03-29T11:08:39.503 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-29T11:08:39.503 INFO:teuthology.orchestra.run.smithi170.stdout:====================================================================================== 2024-03-29T11:08:39.503 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-29T11:08:39.503 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:08:39.503 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-29T11:08:39.504 INFO:teuthology.orchestra.run.smithi170.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:08:39.504 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:08:39.504 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:08:39.504 INFO:teuthology.orchestra.run.smithi170.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:08:39.504 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:08:39.504 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:08:39.504 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:08:39.504 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:08:39.504 INFO:teuthology.orchestra.run.smithi170.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:08:39.504 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:08:39.504 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:08:39.504 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:39.504 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-29T11:08:39.505 INFO:teuthology.orchestra.run.smithi170.stdout:====================================================================================== 2024-03-29T11:08:39.505 INFO:teuthology.orchestra.run.smithi170.stdout:Install 12 Packages 2024-03-29T11:08:39.505 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:39.505 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 6.9 M 2024-03-29T11:08:39.505 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 26 M 2024-03-29T11:08:39.506 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-29T11:08:40.112 INFO:teuthology.orchestra.run.smithi170.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 173 kB/s | 31 kB 00:00 2024-03-29T11:08:40.212 INFO:teuthology.orchestra.run.smithi170.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 54 kB/s | 15 kB 00:00 2024-03-29T11:08:40.271 INFO:teuthology.orchestra.run.smithi170.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-29T11:08:40.304 INFO:teuthology.orchestra.run.smithi170.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.5 MB/s | 238 kB 00:00 2024-03-29T11:08:40.522 INFO:teuthology.orchestra.run.smithi170.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.4 MB/s | 863 kB 00:00 2024-03-29T11:08:40.555 INFO:teuthology.orchestra.run.smithi170.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.1 MB/s | 548 kB 00:00 2024-03-29T11:08:40.597 INFO:teuthology.orchestra.run.smithi170.stdout:(7/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.7 MB/s | 1.8 MB 00:00 2024-03-29T11:08:40.622 INFO:teuthology.orchestra.run.smithi170.stdout:(8/12): grpc-data-1.46.7-10.el9.noarch.rpm 193 kB/s | 19 kB 00:00 2024-03-29T11:08:40.681 INFO:teuthology.orchestra.run.smithi170.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 530 kB/s | 31 kB 00:00 2024-03-29T11:08:40.723 INFO:teuthology.orchestra.run.smithi170.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 12 MB/s | 2.0 MB 00:00 2024-03-29T11:08:40.765 INFO:teuthology.orchestra.run.smithi170.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 862 kB/s | 144 kB 00:00 2024-03-29T11:08:40.806 INFO:teuthology.orchestra.run.smithi170.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-29T11:08:40.809 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:40.809 INFO:teuthology.orchestra.run.smithi170.stdout:Total 5.3 MB/s | 6.9 MB 00:01 2024-03-29T11:08:40.827 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-29T11:08:40.843 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-29T11:08:40.843 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-29T11:08:41.031 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-29T11:08:41.031 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-29T11:08:41.556 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-29T11:08:41.657 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:08:41.692 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:08:41.747 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:08:41.778 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:08:41.881 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:08:42.010 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:08:42.104 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:08:42.148 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:08:42.189 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:08:42.217 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:08:42.457 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:08:42.485 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:08:43.312 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:08:43.312 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:08:43.312 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:08:43.312 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:08:43.313 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:08:43.313 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:08:43.313 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:08:43.313 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:08:43.313 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:08:43.313 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:08:43.313 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:08:43.313 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:08:43.674 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:08:43.674 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:43.674 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-29T11:08:43.674 INFO:teuthology.orchestra.run.smithi170.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:08:43.674 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:08:43.674 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:08:43.674 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:08:43.674 INFO:teuthology.orchestra.run.smithi170.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:08:43.674 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:08:43.674 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:08:43.675 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:08:43.675 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:08:43.675 INFO:teuthology.orchestra.run.smithi170.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:08:43.675 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:08:43.675 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:08:43.675 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:43.675 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:08:43.954 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:08:44.518 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:11 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:08:44.672 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:08:44.673 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-03-29T11:08:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-29T11:08:44.673 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-03-29T11:08:44.673 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-29T11:08:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:08:44.673 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-29T11:08:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:08:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:08:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:08:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:08:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:08:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:08:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:08:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:08:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:08:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:08:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:08:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:44.674 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-29T11:08:44.674 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-03-29T11:08:44.674 INFO:teuthology.orchestra.run.smithi170.stdout:Install 12 Packages 2024-03-29T11:08:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:44.675 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 37 M 2024-03-29T11:08:44.675 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 209 M 2024-03-29T11:08:44.675 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-29T11:08:45.248 INFO:teuthology.orchestra.run.smithi170.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 162 kB/s | 32 kB 00:00 2024-03-29T11:08:45.306 INFO:teuthology.orchestra.run.smithi170.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 295 kB/s | 17 kB 00:00 2024-03-29T11:08:45.348 INFO:teuthology.orchestra.run.smithi170.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 824 kB/s | 34 kB 00:00 2024-03-29T11:08:45.474 INFO:teuthology.orchestra.run.smithi170.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.1 MB/s | 3.0 MB 00:00 2024-03-29T11:08:45.616 INFO:teuthology.orchestra.run.smithi170.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-03-29T11:08:45.725 INFO:teuthology.orchestra.run.smithi170.stdout:(6/12): ceph-mgr-diskprediction-local-19.0.0-14 11 MB/s | 7.4 MB 00:00 2024-03-29T11:08:45.792 INFO:teuthology.orchestra.run.smithi170.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 4.8 MB 00:00 2024-03-29T11:08:45.817 INFO:teuthology.orchestra.run.smithi170.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.1 MB/s | 217 kB 00:00 2024-03-29T11:08:45.967 INFO:teuthology.orchestra.run.smithi170.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-29T11:08:46.018 INFO:teuthology.orchestra.run.smithi170.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.0 MB/s | 803 kB 00:00 2024-03-29T11:08:46.268 INFO:teuthology.orchestra.run.smithi170.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-03-29T11:08:46.426 INFO:teuthology.orchestra.run.smithi170.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 535 kB/s | 245 kB 00:00 2024-03-29T11:08:46.429 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:46.429 INFO:teuthology.orchestra.run.smithi170.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-29T11:08:46.470 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-29T11:08:46.494 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-29T11:08:46.495 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-29T11:08:46.859 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-29T11:08:46.859 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-29T11:08:47.577 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-29T11:08:47.649 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:08:47.689 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:08:47.781 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:08:47.825 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:08:48.002 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:08:48.033 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:08:48.115 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:08:48.756 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:08:48.870 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:08:50.128 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:08:51.149 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:08:51.175 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:08:53.853 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:08:53.853 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:08:53.853 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:08:53.854 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:08:53.854 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:08:53.854 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:08:53.854 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:08:53.854 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:08:53.854 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:08:53.854 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:08:53.855 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:08:53.855 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:08:54.315 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:08:54.315 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:54.315 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-29T11:08:54.315 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:08:54.315 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:08:54.315 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:08:54.315 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:08:54.315 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:08:54.316 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:08:54.316 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:08:54.316 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:08:54.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:08:54.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:08:54.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:08:54.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:08:54.316 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:54.316 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:08:54.738 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:08:55.312 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:08:55.465 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:08:55.466 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2024-03-29T11:08:55.466 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-29T11:08:55.466 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2024-03-29T11:08:55.466 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-29T11:08:55.466 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:08:55.466 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-29T11:08:55.466 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:08:55.466 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:08:55.467 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:08:55.467 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:08:55.467 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:08:55.467 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:08:55.467 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:08:55.467 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:08:55.467 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:08:55.467 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:08:55.467 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:08:55.467 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:08:55.467 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:55.467 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-29T11:08:55.467 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2024-03-29T11:08:55.467 INFO:teuthology.orchestra.run.smithi170.stdout:Install 13 Packages 2024-03-29T11:08:55.468 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:08:55.468 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 2.0 M 2024-03-29T11:08:55.468 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 26 M 2024-03-29T11:08:55.468 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-29T11:08:55.943 INFO:teuthology.orchestra.run.smithi170.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 201 kB/s | 24 kB 00:00 2024-03-29T11:08:55.968 INFO:teuthology.orchestra.run.smithi170.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 124 kB/s | 18 kB 00:00 2024-03-29T11:08:56.018 INFO:teuthology.orchestra.run.smithi170.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:08:56.043 INFO:teuthology.orchestra.run.smithi170.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:08:56.068 INFO:teuthology.orchestra.run.smithi170.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:08:56.094 INFO:teuthology.orchestra.run.smithi170.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2024-03-29T11:08:56.152 INFO:teuthology.orchestra.run.smithi170.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 158 kB/s | 52 kB 00:00 2024-03-29T11:08:56.294 INFO:teuthology.orchestra.run.smithi170.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 143 kB/s | 32 kB 00:00 2024-03-29T11:08:56.319 INFO:teuthology.orchestra.run.smithi170.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 66 kB/s | 15 kB 00:00 2024-03-29T11:08:56.487 INFO:teuthology.orchestra.run.smithi170.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 354 kB/s | 59 kB 00:00 2024-03-29T11:08:56.516 INFO:teuthology.orchestra.run.smithi170.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 557 kB/s | 203 kB 00:00 2024-03-29T11:08:56.646 INFO:teuthology.orchestra.run.smithi170.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 566 kB/s | 90 kB 00:00 2024-03-29T11:08:56.804 INFO:teuthology.orchestra.run.smithi170.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.0 MB/s | 1.0 MB 00:00 2024-03-29T11:08:56.806 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:56.806 INFO:teuthology.orchestra.run.smithi170.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-29T11:08:56.807 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-29T11:08:56.819 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-29T11:08:56.820 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-29T11:08:56.911 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-29T11:08:56.911 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-29T11:08:57.422 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-29T11:08:57.591 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:08:57.646 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:08:57.710 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:08:57.748 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:08:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:08:57.890 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:08:58.000 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:08:58.050 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:08:58.765 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:08:58.822 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:08:58.855 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:08:58.904 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:08:58.926 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:08:59.740 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:08:59.740 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:08:59.741 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:08:59.741 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:08:59.741 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:08:59.741 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:08:59.741 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:08:59.741 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:08:59.741 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:08:59.741 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:08:59.741 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:08:59.741 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:08:59.741 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:09:00.096 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:09:00.096 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:00.096 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-29T11:09:00.096 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:09:00.096 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:09:00.096 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:09:00.096 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:09:00.096 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:09:00.097 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:09:00.097 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:09:00.097 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:09:00.097 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:09:00.097 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:09:00.097 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:09:00.097 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:09:00.097 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:09:00.097 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:00.097 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:09:00.353 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:09:00.924 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:27 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:09:01.078 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:09:01.079 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2024-03-29T11:09:01.079 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-29T11:09:01.079 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2024-03-29T11:09:01.079 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-29T11:09:01.079 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:09:01.079 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-29T11:09:01.079 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:09:01.079 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:09:01.080 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:09:01.080 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:09:01.080 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:09:01.080 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:01.080 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-29T11:09:01.080 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2024-03-29T11:09:01.080 INFO:teuthology.orchestra.run.smithi170.stdout:Install 6 Packages 2024-03-29T11:09:01.080 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:01.081 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 6.8 M 2024-03-29T11:09:01.081 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 33 M 2024-03-29T11:09:01.081 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-29T11:09:01.482 INFO:teuthology.orchestra.run.smithi170.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 604 kB/s | 138 kB 00:00 2024-03-29T11:09:01.582 INFO:teuthology.orchestra.run.smithi170.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 756 kB/s | 248 kB 00:00 2024-03-29T11:09:01.658 INFO:teuthology.orchestra.run.smithi170.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-03-29T11:09:01.783 INFO:teuthology.orchestra.run.smithi170.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 290 kB/s | 58 kB 00:00 2024-03-29T11:09:01.799 INFO:teuthology.orchestra.run.smithi170.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 360 kB/s | 51 kB 00:00 2024-03-29T11:09:01.958 INFO:teuthology.orchestra.run.smithi170.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T11:09:01.961 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:01.961 INFO:teuthology.orchestra.run.smithi170.stdout:Total 7.7 MB/s | 6.8 MB 00:00 2024-03-29T11:09:01.975 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-29T11:09:01.986 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-29T11:09:01.986 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-29T11:09:02.074 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-29T11:09:02.075 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-29T11:09:02.400 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-29T11:09:02.564 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:09:02.618 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:09:03.134 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:09:03.212 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:09:03.274 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:09:03.297 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:09:04.147 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:09:04.147 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:09:04.147 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:09:04.147 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:09:04.147 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:09:04.147 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:09:04.435 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:09:04.436 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:04.436 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-29T11:09:04.436 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:09:04.436 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:09:04.436 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:09:04.436 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:09:04.436 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:09:04.436 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:09:04.436 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:04.436 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:09:04.689 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-fuse 2024-03-29T11:09:05.249 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:09:05.397 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:09:05.398 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:05.398 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-29T11:09:05.398 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:05.398 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-29T11:09:05.398 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:09:05.398 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-29T11:09:05.398 INFO:teuthology.orchestra.run.smithi170.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:09:05.398 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:05.398 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-29T11:09:05.399 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:05.399 INFO:teuthology.orchestra.run.smithi170.stdout:Install 2 Packages 2024-03-29T11:09:05.399 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:05.399 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 964 k 2024-03-29T11:09:05.399 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 2.8 M 2024-03-29T11:09:05.399 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-29T11:09:05.560 INFO:teuthology.orchestra.run.smithi170.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 489 kB/s | 79 kB 00:00 2024-03-29T11:09:05.860 INFO:teuthology.orchestra.run.smithi170.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.9 MB/s | 885 kB 00:00 2024-03-29T11:09:05.861 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:05.862 INFO:teuthology.orchestra.run.smithi170.stdout:Total 2.0 MB/s | 964 kB 00:00 2024-03-29T11:09:05.862 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-29T11:09:05.883 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-29T11:09:05.883 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-29T11:09:05.956 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-29T11:09:05.956 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-29T11:09:06.131 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-29T11:09:06.270 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:09:06.336 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:09:07.062 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:09:07.062 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:09:07.402 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:09:07.403 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:07.403 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-29T11:09:07.403 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:09:07.403 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:07.403 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:09:07.663 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-volume 2024-03-29T11:09:08.251 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:09:08.403 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:09:08.404 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:08.404 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-29T11:09:08.404 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:08.404 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-29T11:09:08.404 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:09:08.404 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-29T11:09:08.404 INFO:teuthology.orchestra.run.smithi170.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:09:08.405 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:08.405 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-29T11:09:08.405 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:08.405 INFO:teuthology.orchestra.run.smithi170.stdout:Install 2 Packages 2024-03-29T11:09:08.405 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:08.405 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 332 k 2024-03-29T11:09:08.405 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 1.5 M 2024-03-29T11:09:08.405 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-29T11:09:08.560 INFO:teuthology.orchestra.run.smithi170.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 444 kB/s | 69 kB 00:00 2024-03-29T11:09:08.958 INFO:teuthology.orchestra.run.smithi170.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 477 kB/s | 264 kB 00:00 2024-03-29T11:09:08.959 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:08.959 INFO:teuthology.orchestra.run.smithi170.stdout:Total 599 kB/s | 332 kB 00:00 2024-03-29T11:09:08.960 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-29T11:09:08.968 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-29T11:09:08.968 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-29T11:09:09.020 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-29T11:09:09.021 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-29T11:09:09.168 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-29T11:09:09.341 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:09:09.394 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:09:09.410 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:09:09.410 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:09:09.410 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:09:09.410 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:10.141 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:09:10.471 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:09:10.471 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:10.471 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-29T11:09:10.471 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:09:10.471 INFO:teuthology.orchestra.run.smithi170.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:09:10.471 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:10.472 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:09:10.792 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados-devel 2024-03-29T11:09:11.373 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:09:11.524 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:09:11.525 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:11.525 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2024-03-29T11:09:11.525 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:11.525 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-29T11:09:11.525 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:09:11.525 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:11.525 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-29T11:09:11.525 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:11.525 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-03-29T11:09:11.525 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:11.525 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 127 k 2024-03-29T11:09:11.525 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 456 k 2024-03-29T11:09:11.525 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-29T11:09:11.750 INFO:teuthology.orchestra.run.smithi170.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 566 kB/s | 127 kB 00:00 2024-03-29T11:09:11.750 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:11.750 INFO:teuthology.orchestra.run.smithi170.stdout:Total 562 kB/s | 127 kB 00:00 2024-03-29T11:09:11.751 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-29T11:09:11.756 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-29T11:09:11.756 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-29T11:09:11.811 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-29T11:09:11.812 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-29T11:09:11.955 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-29T11:09:12.049 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:09:12.525 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:09:12.806 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:09:12.806 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:12.806 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-29T11:09:12.806 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:12.806 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:12.806 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:09:13.045 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install libcephfs2 2024-03-29T11:09:13.622 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:09:13.725 INFO:teuthology.orchestra.run.smithi170.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:09:13.782 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:09:13.782 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-03-29T11:09:13.782 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:09:13.842 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install libcephfs-devel 2024-03-29T11:09:14.436 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:09:14.597 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:09:14.597 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:14.597 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2024-03-29T11:09:14.598 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:14.598 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-29T11:09:14.598 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:09:14.598 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:14.598 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-29T11:09:14.598 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:14.598 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-03-29T11:09:14.598 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:14.598 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 31 k 2024-03-29T11:09:14.598 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 140 k 2024-03-29T11:09:14.598 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-29T11:09:14.846 INFO:teuthology.orchestra.run.smithi170.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 126 kB/s | 31 kB 00:00 2024-03-29T11:09:14.847 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:14.847 INFO:teuthology.orchestra.run.smithi170.stdout:Total 125 kB/s | 31 kB 00:00 2024-03-29T11:09:14.847 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-29T11:09:14.851 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-29T11:09:14.851 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-29T11:09:14.884 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-29T11:09:14.885 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-29T11:09:15.007 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-29T11:09:15.104 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:09:15.540 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:09:15.800 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:09:15.800 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:15.800 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-29T11:09:15.801 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:15.801 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:15.801 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:09:15.977 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados2 2024-03-29T11:09:16.543 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:09:16.640 INFO:teuthology.orchestra.run.smithi170.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:09:16.692 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:09:16.692 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-03-29T11:09:16.693 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:09:16.744 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librbd1 2024-03-29T11:09:17.308 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:09:17.405 INFO:teuthology.orchestra.run.smithi170.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:09:17.457 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:09:17.457 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-03-29T11:09:17.457 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:09:17.507 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rados 2024-03-29T11:09:18.076 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:09:18.174 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:09:18.226 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:09:18.227 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-03-29T11:09:18.227 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:09:18.283 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rgw 2024-03-29T11:09:18.849 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:09:18.948 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:09:18.999 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:09:19.000 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-03-29T11:09:19.001 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:09:19.051 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-cephfs 2024-03-29T11:09:19.620 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:09:19.718 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:09:19.770 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:09:19.771 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-03-29T11:09:19.771 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:09:19.827 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rbd 2024-03-29T11:09:20.394 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:09:20.493 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:09:20.545 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:09:20.546 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-03-29T11:09:20.546 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:09:20.600 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-fuse 2024-03-29T11:09:21.160 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:09:21.309 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:09:21.309 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:21.310 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2024-03-29T11:09:21.310 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:21.310 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-29T11:09:21.310 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:09:21.310 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:21.310 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-29T11:09:21.310 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:21.310 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-03-29T11:09:21.310 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:21.310 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 85 k 2024-03-29T11:09:21.310 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 231 k 2024-03-29T11:09:21.310 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-29T11:09:21.515 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 415 kB/s | 85 kB 00:00 2024-03-29T11:09:21.516 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:21.516 INFO:teuthology.orchestra.run.smithi170.stdout:Total 413 kB/s | 85 kB 00:00 2024-03-29T11:09:21.516 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-29T11:09:21.523 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-29T11:09:21.523 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-29T11:09:21.578 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-29T11:09:21.579 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-29T11:09:21.722 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-29T11:09:21.830 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:09:22.296 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:09:22.608 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:09:22.608 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:22.608 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-29T11:09:22.608 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:22.608 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:22.608 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:09:22.860 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-mirror 2024-03-29T11:09:23.440 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:09:23.598 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:09:23.599 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:23.599 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2024-03-29T11:09:23.599 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:23.599 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-29T11:09:23.599 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:09:23.599 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:23.599 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-29T11:09:23.599 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:23.599 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-03-29T11:09:23.600 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:23.600 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 3.1 M 2024-03-29T11:09:23.600 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 13 M 2024-03-29T11:09:23.600 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-29T11:09:23.994 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 7.8 MB/s | 3.1 MB 00:00 2024-03-29T11:09:23.995 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:23.995 INFO:teuthology.orchestra.run.smithi170.stdout:Total 7.8 MB/s | 3.1 MB 00:00 2024-03-29T11:09:23.996 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-29T11:09:24.005 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-29T11:09:24.006 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-29T11:09:24.077 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-29T11:09:24.077 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-29T11:09:24.503 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-29T11:09:24.562 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:09:24.587 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:09:24.587 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:09:24.587 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:09:24.588 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:09:24.588 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:09:24.588 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:25.386 INFO:teuthology.orchestra.run.smithi184.stdout:Extra Packages for Enterprise Linux 143 kB/s | 21 MB 02:28 2024-03-29T11:09:25.771 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:09:25.771 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:25.771 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-29T11:09:25.772 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:25.772 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:25.772 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:09:26.015 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-nbd 2024-03-29T11:09:26.597 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 11:07:33 AM UTC. 2024-03-29T11:09:26.748 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-29T11:09:26.749 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:26.749 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2024-03-29T11:09:26.749 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:26.749 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-29T11:09:26.749 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:09:26.749 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:26.749 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-29T11:09:26.749 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-29T11:09:26.749 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-03-29T11:09:26.749 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:26.750 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 171 k 2024-03-29T11:09:26.750 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 494 k 2024-03-29T11:09:26.750 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-29T11:09:26.974 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 767 kB/s | 171 kB 00:00 2024-03-29T11:09:26.974 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:26.975 INFO:teuthology.orchestra.run.smithi170.stdout:Total 761 kB/s | 171 kB 00:00 2024-03-29T11:09:26.975 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-29T11:09:26.983 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-29T11:09:26.983 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-29T11:09:27.039 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-29T11:09:27.039 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-29T11:09:27.183 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-29T11:09:27.298 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:09:27.738 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:09:28.017 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:09:28.017 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:28.017 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-29T11:09:28.017 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:28.017 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:09:28.017 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-29T11:09:28.295 DEBUG:teuthology.parallel:result is None 2024-03-29T11:09:30.335 INFO:teuthology.orchestra.run.smithi184.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-03-29T11:09:32.957 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:09:32.969 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:09:32.969 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-03-29T11:09:32.970 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:09:32.970 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-03-29T11:09:32.970 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:09:32.970 INFO:teuthology.orchestra.run.smithi184.stdout:Upgrading: 2024-03-29T11:09:32.970 INFO:teuthology.orchestra.run.smithi184.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:09:32.970 INFO:teuthology.orchestra.run.smithi184.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:09:32.970 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-03-29T11:09:32.970 INFO:teuthology.orchestra.run.smithi184.stdout: boost-program-options 2024-03-29T11:09:32.970 INFO:teuthology.orchestra.run.smithi184.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:09:32.970 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:09:32.970 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:09:32.970 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:09:32.971 INFO:teuthology.orchestra.run.smithi184.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:09:32.971 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:09:32.971 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:09:32.971 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:09:32.971 INFO:teuthology.orchestra.run.smithi184.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:09:32.971 INFO:teuthology.orchestra.run.smithi184.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:09:32.971 INFO:teuthology.orchestra.run.smithi184.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:09:32.971 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:09:32.971 INFO:teuthology.orchestra.run.smithi184.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:09:32.971 INFO:teuthology.orchestra.run.smithi184.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:09:32.971 INFO:teuthology.orchestra.run.smithi184.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:09:32.971 INFO:teuthology.orchestra.run.smithi184.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:09:32.971 INFO:teuthology.orchestra.run.smithi184.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:09:32.971 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse 2024-03-29T11:09:32.971 INFO:teuthology.orchestra.run.smithi184.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:09:32.972 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:09:32.972 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:09:32.972 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:09:32.972 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:09:32.972 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:09:32.972 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:09:32.972 INFO:teuthology.orchestra.run.smithi184.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:09:32.972 INFO:teuthology.orchestra.run.smithi184.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:09:32.972 INFO:teuthology.orchestra.run.smithi184.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:09:32.972 INFO:teuthology.orchestra.run.smithi184.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:09:32.972 INFO:teuthology.orchestra.run.smithi184.stdout:Installing weak dependencies: 2024-03-29T11:09:32.972 INFO:teuthology.orchestra.run.smithi184.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:09:32.972 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:09:32.972 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-03-29T11:09:32.972 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:09:32.973 INFO:teuthology.orchestra.run.smithi184.stdout:Install 30 Packages 2024-03-29T11:09:32.973 INFO:teuthology.orchestra.run.smithi184.stdout:Upgrade 2 Packages 2024-03-29T11:09:32.973 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:09:32.973 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 58 M 2024-03-29T11:09:32.973 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-03-29T11:09:33.769 INFO:teuthology.orchestra.run.smithi184.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 7.4 MB/s | 5.3 MB 00:00 2024-03-29T11:09:33.820 INFO:teuthology.orchestra.run.smithi184.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 506 kB/s | 25 kB 00:00 2024-03-29T11:09:33.920 INFO:teuthology.orchestra.run.smithi184.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 7.3 MB/s | 742 kB 00:00 2024-03-29T11:09:34.012 INFO:teuthology.orchestra.run.smithi184.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 5.3 MB/s | 498 kB 00:00 2024-03-29T11:09:34.670 INFO:teuthology.orchestra.run.smithi184.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 6.0 MB/s | 9.7 MB 00:01 2024-03-29T11:09:34.754 INFO:teuthology.orchestra.run.smithi184.stdout:(6/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 6.9 MB/s | 5.1 MB 00:00 2024-03-29T11:09:34.771 INFO:teuthology.orchestra.run.smithi184.stdout:(7/32): python3-ceph-argparse-19.0.0-1439.g5dcb 450 kB/s | 45 kB 00:00 2024-03-29T11:09:34.813 INFO:teuthology.orchestra.run.smithi184.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.2 MB/s | 131 kB 00:00 2024-03-29T11:09:34.838 INFO:teuthology.orchestra.run.smithi184.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.3 MB/s | 162 kB 00:00 2024-03-29T11:09:34.880 INFO:teuthology.orchestra.run.smithi184.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 4.7 MB/s | 320 kB 00:00 2024-03-29T11:09:34.914 INFO:teuthology.orchestra.run.smithi184.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.9 MB/s | 298 kB 00:00 2024-03-29T11:09:34.973 INFO:teuthology.orchestra.run.smithi184.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.1 MB/s | 99 kB 00:00 2024-03-29T11:09:35.039 INFO:teuthology.orchestra.run.smithi184.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 844 kB/s | 106 kB 00:00 2024-03-29T11:09:35.067 INFO:teuthology.orchestra.run.smithi184.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 159 kB 00:00 2024-03-29T11:09:35.082 INFO:teuthology.orchestra.run.smithi184.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-29T11:09:35.282 INFO:teuthology.orchestra.run.smithi184.stdout:(16/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 8.9 MB/s | 20 MB 00:02 2024-03-29T11:09:35.307 INFO:teuthology.orchestra.run.smithi184.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.7 MB/s | 661 kB 00:00 2024-03-29T11:09:35.332 INFO:teuthology.orchestra.run.smithi184.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.1 MB/s | 282 kB 00:00 2024-03-29T11:09:35.349 INFO:teuthology.orchestra.run.smithi184.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 625 kB/s | 41 kB 00:00 2024-03-29T11:09:35.366 INFO:teuthology.orchestra.run.smithi184.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 35 kB 00:00 2024-03-29T11:09:35.392 INFO:teuthology.orchestra.run.smithi184.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.6 MB/s | 220 kB 00:00 2024-03-29T11:09:35.408 INFO:teuthology.orchestra.run.smithi184.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 792 kB/s | 46 kB 00:00 2024-03-29T11:09:35.592 INFO:teuthology.orchestra.run.smithi184.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 142 kB/s | 26 kB 00:00 2024-03-29T11:09:35.642 INFO:teuthology.orchestra.run.smithi184.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-03-29T11:09:35.659 INFO:teuthology.orchestra.run.smithi184.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 729 kB/s | 49 kB 00:00 2024-03-29T11:09:35.709 INFO:teuthology.orchestra.run.smithi184.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-29T11:09:35.776 INFO:teuthology.orchestra.run.smithi184.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-03-29T11:09:35.801 INFO:teuthology.orchestra.run.smithi184.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.1 MB/s | 192 kB 00:00 2024-03-29T11:09:35.935 INFO:teuthology.orchestra.run.smithi184.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-29T11:09:36.036 INFO:teuthology.orchestra.run.smithi184.stdout:(30/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 4.4 MB 00:00 2024-03-29T11:09:36.160 INFO:teuthology.orchestra.run.smithi184.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 9.4 MB/s | 3.4 MB 00:00 2024-03-29T11:09:36.311 INFO:teuthology.orchestra.run.smithi184.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 8.4 MB/s | 3.1 MB 00:00 2024-03-29T11:09:36.317 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:36.317 INFO:teuthology.orchestra.run.smithi184.stdout:Total 17 MB/s | 58 MB 00:03 2024-03-29T11:09:36.317 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-03-29T11:09:36.353 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-03-29T11:09:36.354 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-03-29T11:09:36.816 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-03-29T11:09:36.816 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-03-29T11:09:38.275 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-03-29T11:09:38.361 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:09:38.391 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:09:38.419 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:09:38.448 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:09:38.464 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:09:38.614 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:09:38.669 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:09:39.008 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:09:39.076 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:09:39.152 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:09:39.241 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:09:39.278 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:09:39.332 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:09:39.361 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:09:39.405 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:09:39.474 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:09:39.497 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:09:39.560 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:09:39.595 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:09:39.697 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:09:39.738 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:09:39.779 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:09:39.812 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:09:39.840 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:09:39.873 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:09:40.186 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:09:40.211 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:09:40.302 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:09:40.333 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:09:40.372 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:09:40.830 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:09:40.864 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:09:40.935 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:09:41.003 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:09:41.033 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:09:43.219 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:09:43.255 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:09:43.833 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:09:43.875 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:09:43.948 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:09:43.949 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:09:43.949 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:09:43.949 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:09:43.993 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:09:53.419 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:09:53.419 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /sys 2024-03-29T11:09:53.419 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /proc 2024-03-29T11:09:53.419 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /mnt 2024-03-29T11:09:53.419 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /var/tmp 2024-03-29T11:09:53.419 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /home 2024-03-29T11:09:53.420 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /root 2024-03-29T11:09:53.420 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /tmp 2024-03-29T11:09:53.420 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:09:54.354 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:09:54.377 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:09:54.378 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:09:54.378 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:09:54.378 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:09:54.378 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:09:54.378 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:09:54.380 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:09:54.456 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:09:54.456 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:09:55.617 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:09:55.618 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:09:55.618 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:09:55.618 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:09:55.618 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:09:55.618 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:09:55.618 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:09:55.618 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:09:55.618 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:09:55.618 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:09:55.618 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:09:55.618 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:09:55.618 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:09:55.619 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:09:55.619 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:09:55.619 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:09:55.619 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:09:55.619 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:09:55.619 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:09:55.619 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:09:55.619 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:09:55.619 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:09:55.619 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:09:55.619 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:09:55.619 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:09:55.619 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:09:55.620 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:09:55.620 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:09:55.620 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:09:55.621 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:09:55.621 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:09:55.621 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:09:55.621 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:09:55.621 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:09:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:09:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:09:56.786 INFO:teuthology.orchestra.run.smithi184.stdout:Upgraded: 2024-03-29T11:09:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:56.786 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-03-29T11:09:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:09:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:56.787 INFO:teuthology.orchestra.run.smithi184.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:09:56.787 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:09:56.787 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:09:56.787 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:56.787 INFO:teuthology.orchestra.run.smithi184.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:09:56.787 INFO:teuthology.orchestra.run.smithi184.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:09:56.787 INFO:teuthology.orchestra.run.smithi184.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:09:56.787 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:56.787 INFO:teuthology.orchestra.run.smithi184.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:09:56.787 INFO:teuthology.orchestra.run.smithi184.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:56.787 INFO:teuthology.orchestra.run.smithi184.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:09:56.787 INFO:teuthology.orchestra.run.smithi184.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:09:56.787 INFO:teuthology.orchestra.run.smithi184.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:09:56.787 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:56.787 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:56.788 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:56.788 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:09:56.788 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:56.788 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:56.788 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:56.788 INFO:teuthology.orchestra.run.smithi184.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:09:56.788 INFO:teuthology.orchestra.run.smithi184.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:09:56.788 INFO:teuthology.orchestra.run.smithi184.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:09:56.788 INFO:teuthology.orchestra.run.smithi184.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:09:56.788 INFO:teuthology.orchestra.run.smithi184.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:09:56.788 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:09:56.788 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:09:57.101 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-test 2024-03-29T11:09:57.682 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:27 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:09:57.832 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:09:57.833 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:09:57.833 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-03-29T11:09:57.833 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:09:57.833 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-03-29T11:09:57.833 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:09:57.833 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-03-29T11:09:57.833 INFO:teuthology.orchestra.run.smithi184.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:09:57.833 INFO:teuthology.orchestra.run.smithi184.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:09:57.833 INFO:teuthology.orchestra.run.smithi184.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:09:57.833 INFO:teuthology.orchestra.run.smithi184.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:09:57.833 INFO:teuthology.orchestra.run.smithi184.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:09:57.833 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:09:57.834 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-03-29T11:09:57.834 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:09:57.834 INFO:teuthology.orchestra.run.smithi184.stdout:Install 6 Packages 2024-03-29T11:09:57.834 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:09:57.834 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 48 M 2024-03-29T11:09:57.834 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 204 M 2024-03-29T11:09:57.835 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-03-29T11:09:58.030 INFO:teuthology.orchestra.run.smithi184.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T11:09:58.080 INFO:teuthology.orchestra.run.smithi184.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T11:09:58.105 INFO:teuthology.orchestra.run.smithi184.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-29T11:09:58.247 INFO:teuthology.orchestra.run.smithi184.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 433 kB/s | 163 kB 00:00 2024-03-29T11:09:58.272 INFO:teuthology.orchestra.run.smithi184.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-29T11:10:00.523 INFO:teuthology.orchestra.run.smithi184.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 18 MB/s | 47 MB 00:02 2024-03-29T11:10:00.525 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:00.526 INFO:teuthology.orchestra.run.smithi184.stdout:Total 18 MB/s | 48 MB 00:02 2024-03-29T11:10:00.540 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-03-29T11:10:00.559 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-03-29T11:10:00.559 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-03-29T11:10:00.942 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-03-29T11:10:00.942 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-03-29T11:10:01.533 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-03-29T11:10:01.598 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:10:01.634 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:10:01.666 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:10:01.716 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:10:01.751 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:10:06.267 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:10:06.341 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:10:08.872 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:10:08.872 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:10:08.872 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:10:08.872 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:10:08.872 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:10:08.872 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:10:09.245 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:10:09.245 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:09.245 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-03-29T11:10:09.245 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:09.245 INFO:teuthology.orchestra.run.smithi184.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:10:09.246 INFO:teuthology.orchestra.run.smithi184.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:09.246 INFO:teuthology.orchestra.run.smithi184.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:10:09.246 INFO:teuthology.orchestra.run.smithi184.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:10:09.246 INFO:teuthology.orchestra.run.smithi184.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:10:09.246 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:09.246 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:10:09.499 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph 2024-03-29T11:10:10.093 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:40 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:10:10.255 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:10:10.257 INFO:teuthology.orchestra.run.smithi184.stdout:========================================================================================= 2024-03-29T11:10:10.257 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-03-29T11:10:10.257 INFO:teuthology.orchestra.run.smithi184.stdout:========================================================================================= 2024-03-29T11:10:10.257 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-03-29T11:10:10.257 INFO:teuthology.orchestra.run.smithi184.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:10:10.257 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-03-29T11:10:10.258 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:10:10.258 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:10:10.258 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:10:10.258 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:10:10.258 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:10:10.258 INFO:teuthology.orchestra.run.smithi184.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:10:10.258 INFO:teuthology.orchestra.run.smithi184.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:10:10.258 INFO:teuthology.orchestra.run.smithi184.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:10:10.258 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:10:10.258 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:10:10.258 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:10:10.258 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:10:10.258 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:10:10.259 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:10:10.259 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:10:10.259 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:10:10.259 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:10:10.259 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:10:10.259 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:10:10.259 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:10:10.259 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:10:10.259 INFO:teuthology.orchestra.run.smithi184.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:10:10.259 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:10:10.259 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:10:10.259 INFO:teuthology.orchestra.run.smithi184.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:10:10.259 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:10:10.259 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:10:10.260 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:10:10.260 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:10:10.260 INFO:teuthology.orchestra.run.smithi184.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:10:10.260 INFO:teuthology.orchestra.run.smithi184.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:10:10.260 INFO:teuthology.orchestra.run.smithi184.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:10:10.260 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:10:10.260 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:10:10.260 INFO:teuthology.orchestra.run.smithi184.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:10:10.260 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:10.260 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-03-29T11:10:10.260 INFO:teuthology.orchestra.run.smithi184.stdout:========================================================================================= 2024-03-29T11:10:10.260 INFO:teuthology.orchestra.run.smithi184.stdout:Install 36 Packages 2024-03-29T11:10:10.260 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:10.261 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 30 M 2024-03-29T11:10:10.261 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 107 M 2024-03-29T11:10:10.261 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-03-29T11:10:10.958 INFO:teuthology.orchestra.run.smithi184.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 19 kB/s | 6.6 kB 00:00 2024-03-29T11:10:11.925 INFO:teuthology.orchestra.run.smithi184.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.6 MB/s | 2.1 MB 00:01 2024-03-29T11:10:12.125 INFO:teuthology.orchestra.run.smithi184.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.0 MB/s | 1.5 MB 00:01 2024-03-29T11:10:12.275 INFO:teuthology.orchestra.run.smithi184.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.6 MB/s | 247 kB 00:00 2024-03-29T11:10:12.392 INFO:teuthology.orchestra.run.smithi184.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T11:10:12.434 INFO:teuthology.orchestra.run.smithi184.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:10:12.468 INFO:teuthology.orchestra.run.smithi184.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 949 kB/s | 32 kB 00:00 2024-03-29T11:10:12.501 INFO:teuthology.orchestra.run.smithi184.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:10:12.535 INFO:teuthology.orchestra.run.smithi184.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:10:12.585 INFO:teuthology.orchestra.run.smithi184.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T11:10:12.652 INFO:teuthology.orchestra.run.smithi184.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T11:10:12.686 INFO:teuthology.orchestra.run.smithi184.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T11:10:12.719 INFO:teuthology.orchestra.run.smithi184.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T11:10:12.753 INFO:teuthology.orchestra.run.smithi184.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T11:10:12.920 INFO:teuthology.orchestra.run.smithi184.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 749 kB/s | 125 kB 00:00 2024-03-29T11:10:13.012 INFO:teuthology.orchestra.run.smithi184.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-03-29T11:10:13.445 INFO:teuthology.orchestra.run.smithi184.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 349 kB/s | 151 kB 00:00 2024-03-29T11:10:13.538 INFO:teuthology.orchestra.run.smithi184.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 474 kB/s | 43 kB 00:00 2024-03-29T11:10:13.713 INFO:teuthology.orchestra.run.smithi184.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 982 kB/s | 172 kB 00:00 2024-03-29T11:10:13.871 INFO:teuthology.orchestra.run.smithi184.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.2 MB/s | 358 kB 00:00 2024-03-29T11:10:13.963 INFO:teuthology.orchestra.run.smithi184.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-03-29T11:10:14.072 INFO:teuthology.orchestra.run.smithi184.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-03-29T11:10:14.156 INFO:teuthology.orchestra.run.smithi184.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 278 kB/s | 23 kB 00:00 2024-03-29T11:10:14.239 INFO:teuthology.orchestra.run.smithi184.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 234 kB/s | 19 kB 00:00 2024-03-29T11:10:14.331 INFO:teuthology.orchestra.run.smithi184.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 215 kB/s | 20 kB 00:00 2024-03-29T11:10:14.423 INFO:teuthology.orchestra.run.smithi184.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 506 kB/s | 46 kB 00:00 2024-03-29T11:10:14.515 INFO:teuthology.orchestra.run.smithi184.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 861 kB/s | 79 kB 00:00 2024-03-29T11:10:14.624 INFO:teuthology.orchestra.run.smithi184.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-03-29T11:10:14.724 INFO:teuthology.orchestra.run.smithi184.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-29T11:10:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 981 kB/s | 90 kB 00:00 2024-03-29T11:10:14.916 INFO:teuthology.orchestra.run.smithi184.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 358 kB/s | 36 kB 00:00 2024-03-29T11:10:14.984 INFO:teuthology.orchestra.run.smithi184.stdout:(32/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.2 MB/s | 4.7 MB 00:04 2024-03-29T11:10:15.009 INFO:teuthology.orchestra.run.smithi184.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-29T11:10:15.101 INFO:teuthology.orchestra.run.smithi184.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-03-29T11:10:15.593 INFO:teuthology.orchestra.run.smithi184.stdout:(35/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 4.6 MB/s | 17 MB 00:03 2024-03-29T11:10:15.617 INFO:teuthology.orchestra.run.smithi184.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 674 kB/s | 427 kB 00:00 2024-03-29T11:10:15.622 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:15.622 INFO:teuthology.orchestra.run.smithi184.stdout:Total 5.5 MB/s | 30 MB 00:05 2024-03-29T11:10:15.646 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-03-29T11:10:15.690 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-03-29T11:10:15.690 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-03-29T11:10:16.131 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-03-29T11:10:16.132 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-03-29T11:10:17.461 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-03-29T11:10:17.520 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:10:17.559 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:10:17.589 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:10:17.659 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:10:17.705 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:10:17.738 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:10:17.830 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:10:17.917 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:10:17.957 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:10:18.029 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:10:18.091 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:10:18.129 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:10:18.178 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:10:18.234 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:10:18.420 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:10:18.475 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:10:18.505 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:10:18.539 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:10:18.569 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:10:18.646 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:10:18.680 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:10:18.713 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:10:18.753 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:10:18.797 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:10:18.831 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:10:18.922 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:10:18.962 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:10:19.035 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:10:19.171 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:10:19.305 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:10:19.462 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:10:19.492 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:10:19.517 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:10:19.517 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:19.517 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:10:19.518 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:10:19.518 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:10:19.518 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:20.951 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:10:20.977 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:10:20.978 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:20.978 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:10:20.978 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:10:20.978 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:10:20.979 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:21.419 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:10:21.442 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:10:21.442 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:21.442 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:10:21.443 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:10:21.443 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:10:21.443 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:21.653 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:10:21.676 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:10:21.676 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:21.676 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:10:21.676 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:10:21.676 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:10:21.676 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:21.746 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:10:23.536 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:10:23.536 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:10:23.536 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:10:23.536 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:10:23.536 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:10:23.536 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:10:23.536 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:10:23.536 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:10:23.536 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:10:23.537 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:10:23.537 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:10:23.537 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:10:23.537 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:10:23.537 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:10:23.537 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:10:23.537 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:10:23.537 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:10:23.537 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:10:23.537 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:10:23.537 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:10:23.538 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:10:23.538 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:10:23.538 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:10:23.538 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:10:23.538 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:10:23.538 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:10:23.538 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:10:23.538 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:10:23.539 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:10:23.539 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:10:23.540 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:10:23.540 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:10:23.540 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:10:23.540 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:10:23.540 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:10:23.540 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:10:24.271 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:10:24.271 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:24.271 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-03-29T11:10:24.271 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.271 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.271 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.271 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:24.271 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.271 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.271 INFO:teuthology.orchestra.run.smithi184.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:10:24.272 INFO:teuthology.orchestra.run.smithi184.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:10:24.272 INFO:teuthology.orchestra.run.smithi184.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:10:24.272 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:10:24.272 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:10:24.272 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:10:24.272 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:10:24.272 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:10:24.272 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:10:24.272 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:10:24.272 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:10:24.273 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:10:24.273 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:10:24.273 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:10:24.273 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:10:24.273 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:10:24.273 INFO:teuthology.orchestra.run.smithi184.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:10:24.273 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:10:24.273 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:10:24.273 INFO:teuthology.orchestra.run.smithi184.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:10:24.273 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:10:24.273 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:10:24.273 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:10:24.273 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:10:24.274 INFO:teuthology.orchestra.run.smithi184.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:10:24.274 INFO:teuthology.orchestra.run.smithi184.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:10:24.274 INFO:teuthology.orchestra.run.smithi184.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:10:24.274 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:10:24.274 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:10:24.274 INFO:teuthology.orchestra.run.smithi184.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:10:24.274 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:24.274 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:10:24.592 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-base 2024-03-29T11:10:25.176 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:10:25.273 INFO:teuthology.orchestra.run.smithi184.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:25.325 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:10:25.326 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-03-29T11:10:25.326 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:10:25.401 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install cephadm 2024-03-29T11:10:25.971 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:10:26.124 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:10:26.124 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:10:26.125 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-03-29T11:10:26.125 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:10:26.125 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-03-29T11:10:26.125 INFO:teuthology.orchestra.run.smithi184.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:10:26.125 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:26.125 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-03-29T11:10:26.125 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:10:26.125 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-03-29T11:10:26.125 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:26.125 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 747 k 2024-03-29T11:10:26.125 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 754 k 2024-03-29T11:10:26.125 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-03-29T11:10:26.926 INFO:teuthology.orchestra.run.smithi184.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 933 kB/s | 747 kB 00:00 2024-03-29T11:10:26.927 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:26.927 INFO:teuthology.orchestra.run.smithi184.stdout:Total 932 kB/s | 747 kB 00:00 2024-03-29T11:10:26.927 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-03-29T11:10:26.933 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-03-29T11:10:26.933 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-03-29T11:10:26.939 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-03-29T11:10:26.939 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-03-29T11:10:27.014 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-03-29T11:10:27.522 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:10:27.607 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:10:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:10:28.304 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:10:28.305 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:28.305 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-03-29T11:10:28.305 INFO:teuthology.orchestra.run.smithi184.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:28.305 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:28.305 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:10:28.522 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:10:29.088 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:10:29.236 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:10:29.237 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:10:29.237 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2024-03-29T11:10:29.237 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:10:29.237 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-03-29T11:10:29.237 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:10:29.237 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:29.237 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-03-29T11:10:29.237 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:10:29.237 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-03-29T11:10:29.238 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:29.238 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 145 k 2024-03-29T11:10:29.238 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 436 k 2024-03-29T11:10:29.238 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-03-29T11:10:29.565 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 444 kB/s | 145 kB 00:00 2024-03-29T11:10:29.566 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:29.566 INFO:teuthology.orchestra.run.smithi184.stdout:Total 442 kB/s | 145 kB 00:00 2024-03-29T11:10:29.566 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-03-29T11:10:29.573 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-03-29T11:10:29.573 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-03-29T11:10:29.625 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-03-29T11:10:29.626 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-03-29T11:10:29.760 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-03-29T11:10:29.841 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:10:29.865 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:10:29.865 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:29.865 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:10:29.865 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:30.790 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:10:30.790 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:30.790 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-03-29T11:10:30.790 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:30.790 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:30.790 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:10:31.083 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr 2024-03-29T11:10:31.662 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:10:31.760 INFO:teuthology.orchestra.run.smithi184.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:31.813 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:10:31.814 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-03-29T11:10:31.814 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:10:31.879 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:10:32.442 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:10:32.592 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:10:32.593 INFO:teuthology.orchestra.run.smithi184.stdout:====================================================================================== 2024-03-29T11:10:32.593 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-03-29T11:10:32.593 INFO:teuthology.orchestra.run.smithi184.stdout:====================================================================================== 2024-03-29T11:10:32.593 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-03-29T11:10:32.593 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:10:32.593 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-03-29T11:10:32.593 INFO:teuthology.orchestra.run.smithi184.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:10:32.593 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:10:32.593 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:10:32.593 INFO:teuthology.orchestra.run.smithi184.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:10:32.593 INFO:teuthology.orchestra.run.smithi184.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:10:32.594 INFO:teuthology.orchestra.run.smithi184.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:10:32.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:10:32.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:10:32.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:10:32.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:10:32.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:10:32.594 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:32.594 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-03-29T11:10:32.594 INFO:teuthology.orchestra.run.smithi184.stdout:====================================================================================== 2024-03-29T11:10:32.594 INFO:teuthology.orchestra.run.smithi184.stdout:Install 12 Packages 2024-03-29T11:10:32.594 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:32.595 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 6.9 M 2024-03-29T11:10:32.595 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 26 M 2024-03-29T11:10:32.595 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-03-29T11:10:32.960 INFO:teuthology.orchestra.run.smithi184.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 146 kB/s | 31 kB 00:00 2024-03-29T11:10:32.985 INFO:teuthology.orchestra.run.smithi184.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 64 kB/s | 15 kB 00:00 2024-03-29T11:10:33.135 INFO:teuthology.orchestra.run.smithi184.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 4.6 MB/s | 1.8 MB 00:00 2024-03-29T11:10:33.169 INFO:teuthology.orchestra.run.smithi184.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-29T11:10:33.194 INFO:teuthology.orchestra.run.smithi184.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 238 kB 00:00 2024-03-29T11:10:33.436 INFO:teuthology.orchestra.run.smithi184.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.8 MB/s | 863 kB 00:00 2024-03-29T11:10:33.686 INFO:teuthology.orchestra.run.smithi184.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 39 kB/s | 19 kB 00:00 2024-03-29T11:10:33.995 INFO:teuthology.orchestra.run.smithi184.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 468 kB/s | 144 kB 00:00 2024-03-29T11:10:34.037 INFO:teuthology.orchestra.run.smithi184.stdout:(9/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 651 kB/s | 548 kB 00:00 2024-03-29T11:10:34.112 INFO:teuthology.orchestra.run.smithi184.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 263 kB/s | 31 kB 00:00 2024-03-29T11:10:34.162 INFO:teuthology.orchestra.run.smithi184.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-29T11:10:34.379 INFO:teuthology.orchestra.run.smithi184.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.2 MB/s | 2.0 MB 00:00 2024-03-29T11:10:34.384 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:34.384 INFO:teuthology.orchestra.run.smithi184.stdout:Total 3.8 MB/s | 6.9 MB 00:01 2024-03-29T11:10:34.403 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-03-29T11:10:34.419 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-03-29T11:10:34.420 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-03-29T11:10:34.601 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-03-29T11:10:34.601 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-03-29T11:10:35.111 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-03-29T11:10:35.196 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:10:35.240 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:10:35.287 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:10:35.325 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:10:35.436 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:10:35.558 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:10:35.653 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:10:35.738 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:10:35.766 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:10:35.798 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:10:36.042 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:10:36.073 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:10:36.838 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:10:36.838 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:10:36.838 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:10:36.838 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:10:36.838 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:10:36.838 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:10:36.838 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:10:36.839 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:10:36.839 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:10:36.839 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:10:36.839 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:10:36.839 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:10:37.183 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:10:37.183 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:37.183 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-03-29T11:10:37.183 INFO:teuthology.orchestra.run.smithi184.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:10:37.183 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:37.183 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:37.183 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:37.183 INFO:teuthology.orchestra.run.smithi184.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:10:37.183 INFO:teuthology.orchestra.run.smithi184.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:10:37.183 INFO:teuthology.orchestra.run.smithi184.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:10:37.184 INFO:teuthology.orchestra.run.smithi184.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:10:37.184 INFO:teuthology.orchestra.run.smithi184.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:10:37.184 INFO:teuthology.orchestra.run.smithi184.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:10:37.184 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:10:37.184 INFO:teuthology.orchestra.run.smithi184.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:10:37.184 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:37.184 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:10:37.441 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:10:38.008 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:08 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:10:38.161 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:10:38.162 INFO:teuthology.orchestra.run.smithi184.stdout:============================================================================================ 2024-03-29T11:10:38.162 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-03-29T11:10:38.162 INFO:teuthology.orchestra.run.smithi184.stdout:============================================================================================ 2024-03-29T11:10:38.162 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-03-29T11:10:38.162 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:10:38.163 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-03-29T11:10:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:10:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:10:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:10:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:10:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:10:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:10:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:10:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:10:38.164 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:10:38.164 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:10:38.164 INFO:teuthology.orchestra.run.smithi184.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:10:38.164 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:38.164 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-03-29T11:10:38.164 INFO:teuthology.orchestra.run.smithi184.stdout:============================================================================================ 2024-03-29T11:10:38.164 INFO:teuthology.orchestra.run.smithi184.stdout:Install 12 Packages 2024-03-29T11:10:38.164 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:38.165 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 37 M 2024-03-29T11:10:38.165 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 209 M 2024-03-29T11:10:38.165 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-03-29T11:10:38.627 INFO:teuthology.orchestra.run.smithi184.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 238 kB/s | 32 kB 00:00 2024-03-29T11:10:38.652 INFO:teuthology.orchestra.run.smithi184.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 719 kB/s | 17 kB 00:00 2024-03-29T11:10:38.686 INFO:teuthology.orchestra.run.smithi184.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T11:10:38.736 INFO:teuthology.orchestra.run.smithi184.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-29T11:10:38.928 INFO:teuthology.orchestra.run.smithi184.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-29T11:10:38.995 INFO:teuthology.orchestra.run.smithi184.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-29T11:10:39.020 INFO:teuthology.orchestra.run.smithi184.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T11:10:39.329 INFO:teuthology.orchestra.run.smithi184.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-29T11:10:39.479 INFO:teuthology.orchestra.run.smithi184.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-03-29T11:10:39.504 INFO:teuthology.orchestra.run.smithi184.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-29T11:10:39.563 INFO:teuthology.orchestra.run.smithi184.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-03-29T11:10:40.405 INFO:teuthology.orchestra.run.smithi184.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 3.9 MB/s | 7.4 MB 00:01 2024-03-29T11:10:40.406 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:40.407 INFO:teuthology.orchestra.run.smithi184.stdout:Total 17 MB/s | 37 MB 00:02 2024-03-29T11:10:40.446 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-03-29T11:10:40.469 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-03-29T11:10:40.470 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-03-29T11:10:40.831 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-03-29T11:10:40.831 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-03-29T11:10:41.529 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-03-29T11:10:41.594 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:10:41.634 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:10:41.728 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:10:41.770 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:10:41.947 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:10:41.978 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:10:42.064 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:10:42.708 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:10:42.815 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:10:44.060 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:10:45.088 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:10:45.110 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:10:47.621 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:10:47.621 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:10:47.621 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:10:47.622 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:10:47.622 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:10:47.622 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:10:47.622 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:10:47.622 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:10:47.622 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:10:47.622 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:10:47.622 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:10:47.622 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:10:48.077 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:10:48.077 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:48.077 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-03-29T11:10:48.077 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:48.078 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:10:48.078 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:10:48.078 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:10:48.078 INFO:teuthology.orchestra.run.smithi184.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:10:48.078 INFO:teuthology.orchestra.run.smithi184.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:10:48.078 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:10:48.078 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:10:48.078 INFO:teuthology.orchestra.run.smithi184.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:10:48.078 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:10:48.078 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:10:48.078 INFO:teuthology.orchestra.run.smithi184.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:10:48.078 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:48.078 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:10:48.466 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:10:49.039 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:19 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:10:49.190 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:10:49.191 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================== 2024-03-29T11:10:49.191 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-03-29T11:10:49.191 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================== 2024-03-29T11:10:49.191 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-03-29T11:10:49.191 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:10:49.191 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-03-29T11:10:49.191 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:10:49.191 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:10:49.191 INFO:teuthology.orchestra.run.smithi184.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:10:49.192 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:10:49.192 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:10:49.192 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:10:49.192 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:10:49.192 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:10:49.192 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:10:49.192 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:10:49.192 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:10:49.192 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:10:49.192 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:49.192 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-03-29T11:10:49.192 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================== 2024-03-29T11:10:49.192 INFO:teuthology.orchestra.run.smithi184.stdout:Install 13 Packages 2024-03-29T11:10:49.192 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:49.193 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 2.0 M 2024-03-29T11:10:49.193 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 26 M 2024-03-29T11:10:49.193 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-03-29T11:10:49.461 INFO:teuthology.orchestra.run.smithi184.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 206 kB/s | 24 kB 00:00 2024-03-29T11:10:49.486 INFO:teuthology.orchestra.run.smithi184.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 126 kB/s | 18 kB 00:00 2024-03-29T11:10:49.536 INFO:teuthology.orchestra.run.smithi184.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:10:49.553 INFO:teuthology.orchestra.run.smithi184.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-29T11:10:49.578 INFO:teuthology.orchestra.run.smithi184.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2024-03-29T11:10:49.604 INFO:teuthology.orchestra.run.smithi184.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2024-03-29T11:10:49.629 INFO:teuthology.orchestra.run.smithi184.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 182 kB/s | 52 kB 00:00 2024-03-29T11:10:49.688 INFO:teuthology.orchestra.run.smithi184.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 179 kB/s | 15 kB 00:00 2024-03-29T11:10:49.713 INFO:teuthology.orchestra.run.smithi184.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 240 kB/s | 32 kB 00:00 2024-03-29T11:10:49.755 INFO:teuthology.orchestra.run.smithi184.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.6 MB/s | 203 kB 00:00 2024-03-29T11:10:49.774 INFO:teuthology.orchestra.run.smithi184.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-29T11:10:49.797 INFO:teuthology.orchestra.run.smithi184.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 2.1 MB/s | 90 kB 00:00 2024-03-29T11:10:49.905 INFO:teuthology.orchestra.run.smithi184.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.7 MB/s | 1.0 MB 00:00 2024-03-29T11:10:49.907 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:49.907 INFO:teuthology.orchestra.run.smithi184.stdout:Total 2.9 MB/s | 2.0 MB 00:00 2024-03-29T11:10:49.907 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-03-29T11:10:49.920 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-03-29T11:10:49.921 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-03-29T11:10:50.010 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-03-29T11:10:50.011 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-03-29T11:10:50.536 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-03-29T11:10:50.719 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:10:50.773 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:10:50.828 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:10:50.866 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:10:50.906 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:10:51.001 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:10:51.109 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:10:51.167 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:10:51.880 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:10:51.930 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:10:51.964 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:10:52.013 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:10:52.034 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:10:52.772 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:10:52.772 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:10:52.772 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:10:52.772 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:10:52.772 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:10:52.772 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:10:52.772 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:10:52.772 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:10:52.772 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:10:52.772 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:10:52.773 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:10:52.773 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:10:52.773 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:10:53.131 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:10:53.131 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:53.131 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-03-29T11:10:53.131 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:53.131 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:10:53.131 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:10:53.131 INFO:teuthology.orchestra.run.smithi184.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:10:53.131 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:10:53.132 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:10:53.132 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:10:53.132 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:10:53.132 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:10:53.132 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:10:53.132 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:10:53.132 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:10:53.132 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:10:53.132 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:53.132 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:10:53.385 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:10:53.962 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:23 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:10:54.115 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:10:54.116 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================== 2024-03-29T11:10:54.116 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-03-29T11:10:54.116 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================== 2024-03-29T11:10:54.116 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-03-29T11:10:54.116 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:10:54.116 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-03-29T11:10:54.116 INFO:teuthology.orchestra.run.smithi184.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:10:54.116 INFO:teuthology.orchestra.run.smithi184.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:10:54.116 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:10:54.117 INFO:teuthology.orchestra.run.smithi184.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:10:54.117 INFO:teuthology.orchestra.run.smithi184.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:10:54.117 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:54.117 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-03-29T11:10:54.117 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================== 2024-03-29T11:10:54.117 INFO:teuthology.orchestra.run.smithi184.stdout:Install 6 Packages 2024-03-29T11:10:54.117 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:54.118 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 6.8 M 2024-03-29T11:10:54.118 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 33 M 2024-03-29T11:10:54.118 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-03-29T11:10:54.690 INFO:teuthology.orchestra.run.smithi184.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-29T11:10:54.782 INFO:teuthology.orchestra.run.smithi184.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 551 kB/s | 138 kB 00:00 2024-03-29T11:10:54.873 INFO:teuthology.orchestra.run.smithi184.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-29T11:10:55.157 INFO:teuthology.orchestra.run.smithi184.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 164 kB/s | 58 kB 00:00 2024-03-29T11:10:55.191 INFO:teuthology.orchestra.run.smithi184.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 174 kB/s | 51 kB 00:00 2024-03-29T11:10:55.216 INFO:teuthology.orchestra.run.smithi184.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-03-29T11:10:55.219 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:55.220 INFO:teuthology.orchestra.run.smithi184.stdout:Total 6.2 MB/s | 6.8 MB 00:01 2024-03-29T11:10:55.232 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-03-29T11:10:55.243 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-03-29T11:10:55.243 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-03-29T11:10:55.331 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-03-29T11:10:55.332 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-03-29T11:10:55.626 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-03-29T11:10:55.790 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:10:55.843 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:10:56.363 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:10:56.445 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:10:56.514 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:10:56.538 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:10:57.319 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:10:57.319 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:10:57.319 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:10:57.319 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:10:57.320 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:10:57.320 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:10:57.630 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:10:57.631 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:57.631 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-03-29T11:10:57.631 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:57.631 INFO:teuthology.orchestra.run.smithi184.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:10:57.631 INFO:teuthology.orchestra.run.smithi184.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:10:57.631 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:10:57.631 INFO:teuthology.orchestra.run.smithi184.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:10:57.631 INFO:teuthology.orchestra.run.smithi184.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:10:57.631 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:57.631 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:10:57.898 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-fuse 2024-03-29T11:10:58.464 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:10:58.613 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:10:58.614 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:10:58.614 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-03-29T11:10:58.615 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:10:58.615 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-03-29T11:10:58.615 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:10:58.616 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-03-29T11:10:58.616 INFO:teuthology.orchestra.run.smithi184.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:10:58.616 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:58.616 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-03-29T11:10:58.616 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:10:58.616 INFO:teuthology.orchestra.run.smithi184.stdout:Install 2 Packages 2024-03-29T11:10:58.616 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:10:58.617 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 964 k 2024-03-29T11:10:58.617 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 2.8 M 2024-03-29T11:10:58.617 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-03-29T11:10:58.744 INFO:teuthology.orchestra.run.smithi184.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 609 kB/s | 79 kB 00:00 2024-03-29T11:10:59.061 INFO:teuthology.orchestra.run.smithi184.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.9 MB/s | 885 kB 00:00 2024-03-29T11:10:59.062 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:59.063 INFO:teuthology.orchestra.run.smithi184.stdout:Total 2.1 MB/s | 964 kB 00:00 2024-03-29T11:10:59.063 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-03-29T11:10:59.085 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-03-29T11:10:59.085 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-03-29T11:10:59.158 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-03-29T11:10:59.158 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-03-29T11:10:59.340 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-03-29T11:10:59.478 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:10:59.545 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:11:00.229 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:11:00.229 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:11:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:11:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:00.556 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-03-29T11:11:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:11:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:00.556 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:11:00.869 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-volume 2024-03-29T11:11:01.455 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:11:01.609 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:11:01.610 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:11:01.610 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-03-29T11:11:01.610 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:11:01.610 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-03-29T11:11:01.610 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:11:01.610 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-03-29T11:11:01.610 INFO:teuthology.orchestra.run.smithi184.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:11:01.610 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:01.610 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-03-29T11:11:01.611 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:11:01.611 INFO:teuthology.orchestra.run.smithi184.stdout:Install 2 Packages 2024-03-29T11:11:01.611 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:01.611 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 332 k 2024-03-29T11:11:01.611 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 1.5 M 2024-03-29T11:11:01.612 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-03-29T11:11:01.740 INFO:teuthology.orchestra.run.smithi184.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 535 kB/s | 69 kB 00:00 2024-03-29T11:11:01.999 INFO:teuthology.orchestra.run.smithi184.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 681 kB/s | 264 kB 00:00 2024-03-29T11:11:02.000 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:02.000 INFO:teuthology.orchestra.run.smithi184.stdout:Total 854 kB/s | 332 kB 00:00 2024-03-29T11:11:02.000 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-03-29T11:11:02.009 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-03-29T11:11:02.009 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-03-29T11:11:02.048 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-03-29T11:11:02.048 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-03-29T11:11:02.199 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-03-29T11:11:02.390 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:11:02.435 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:11:02.451 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:11:02.451 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:11:02.451 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:11:02.451 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:03.195 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:11:03.542 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:11:03.542 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:03.542 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-03-29T11:11:03.542 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:11:03.542 INFO:teuthology.orchestra.run.smithi184.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:11:03.542 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:03.542 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:11:03.838 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librados-devel 2024-03-29T11:11:04.427 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:11:04.578 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:11:04.579 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:11:04.579 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2024-03-29T11:11:04.579 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:11:04.579 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-03-29T11:11:04.579 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:11:04.580 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:04.580 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-03-29T11:11:04.580 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:11:04.580 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-03-29T11:11:04.580 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:04.580 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 127 k 2024-03-29T11:11:04.580 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 456 k 2024-03-29T11:11:04.580 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-03-29T11:11:04.906 INFO:teuthology.orchestra.run.smithi184.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 391 kB/s | 127 kB 00:00 2024-03-29T11:11:04.906 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:04.906 INFO:teuthology.orchestra.run.smithi184.stdout:Total 389 kB/s | 127 kB 00:00 2024-03-29T11:11:04.906 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-03-29T11:11:04.911 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-03-29T11:11:04.911 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-03-29T11:11:04.966 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-03-29T11:11:04.966 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-03-29T11:11:05.112 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-03-29T11:11:05.211 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:05.678 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:05.993 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:05.993 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:05.993 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-03-29T11:11:05.994 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:11:05.994 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:05.994 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:11:06.240 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install libcephfs2 2024-03-29T11:11:06.816 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:11:06.920 INFO:teuthology.orchestra.run.smithi184.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:06.975 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:11:06.976 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-03-29T11:11:06.976 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:11:07.051 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install libcephfs-devel 2024-03-29T11:11:07.616 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:11:07.768 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:11:07.769 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:11:07.769 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2024-03-29T11:11:07.769 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:11:07.769 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-03-29T11:11:07.769 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:11:07.769 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:07.769 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-03-29T11:11:07.769 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:11:07.769 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-03-29T11:11:07.769 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:07.769 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 31 k 2024-03-29T11:11:07.770 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 140 k 2024-03-29T11:11:07.770 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-03-29T11:11:08.082 INFO:teuthology.orchestra.run.smithi184.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 100 kB/s | 31 kB 00:00 2024-03-29T11:11:08.083 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:08.083 INFO:teuthology.orchestra.run.smithi184.stdout:Total 99 kB/s | 31 kB 00:00 2024-03-29T11:11:08.083 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-03-29T11:11:08.087 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-03-29T11:11:08.087 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-03-29T11:11:08.117 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-03-29T11:11:08.118 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-03-29T11:11:08.248 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-03-29T11:11:08.377 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:08.719 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:09.001 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:09.001 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:09.001 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-03-29T11:11:09.001 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:11:09.001 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:09.001 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:11:09.172 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librados2 2024-03-29T11:11:09.732 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:11:09.829 INFO:teuthology.orchestra.run.smithi184.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:09.880 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:11:09.881 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-03-29T11:11:09.881 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:11:09.932 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librbd1 2024-03-29T11:11:10.494 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:11:10.592 INFO:teuthology.orchestra.run.smithi184.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:10.643 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:11:10.645 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-03-29T11:11:10.645 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:11:10.701 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rados 2024-03-29T11:11:11.267 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:11:11.369 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:11.422 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:11:11.423 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-03-29T11:11:11.423 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:11:11.480 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rgw 2024-03-29T11:11:12.045 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:11:12.143 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:12.195 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:11:12.196 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-03-29T11:11:12.196 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:11:12.259 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-cephfs 2024-03-29T11:11:12.831 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:11:12.929 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:12.981 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:11:12.982 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-03-29T11:11:12.982 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:11:13.031 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rbd 2024-03-29T11:11:13.594 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:11:13.691 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:13.743 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:11:13.744 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-03-29T11:11:13.744 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:11:13.797 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-fuse 2024-03-29T11:11:14.357 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:11:14.505 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:11:14.505 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:11:14.505 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-03-29T11:11:14.506 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:11:14.506 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-03-29T11:11:14.506 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:11:14.506 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:14.506 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-03-29T11:11:14.506 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:11:14.506 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-03-29T11:11:14.506 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:14.506 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 85 k 2024-03-29T11:11:14.506 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 231 k 2024-03-29T11:11:14.506 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-03-29T11:11:14.810 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 281 kB/s | 85 kB 00:00 2024-03-29T11:11:14.810 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:14.811 INFO:teuthology.orchestra.run.smithi184.stdout:Total 280 kB/s | 85 kB 00:00 2024-03-29T11:11:14.811 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-03-29T11:11:14.818 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-03-29T11:11:14.818 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-03-29T11:11:14.873 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-03-29T11:11:14.873 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-03-29T11:11:15.011 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-03-29T11:11:15.100 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:15.476 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:15.793 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:15.793 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:15.794 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-03-29T11:11:15.794 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:11:15.794 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:15.794 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:11:16.044 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-mirror 2024-03-29T11:11:16.640 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:11:16.801 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:11:16.802 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:11:16.802 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2024-03-29T11:11:16.802 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:11:16.802 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-03-29T11:11:16.802 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:11:16.802 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:16.802 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-03-29T11:11:16.802 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:11:16.802 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-03-29T11:11:16.802 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:16.802 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 3.1 M 2024-03-29T11:11:16.803 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 13 M 2024-03-29T11:11:16.803 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-03-29T11:11:18.121 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.3 MB/s | 3.1 MB 00:01 2024-03-29T11:11:18.122 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:18.123 INFO:teuthology.orchestra.run.smithi184.stdout:Total 2.3 MB/s | 3.1 MB 00:01 2024-03-29T11:11:18.123 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-03-29T11:11:18.133 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-03-29T11:11:18.134 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-03-29T11:11:18.209 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-03-29T11:11:18.210 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-03-29T11:11:18.632 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-03-29T11:11:18.689 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:18.714 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:18.714 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:11:18.715 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:11:18.715 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:11:18.715 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:11:18.715 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:19.893 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:19.893 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:19.894 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-03-29T11:11:19.894 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:11:19.894 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:19.894 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:11:20.141 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-nbd 2024-03-29T11:11:20.722 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 11:09:30 AM UTC. 2024-03-29T11:11:20.872 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-29T11:11:20.872 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:11:20.873 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-03-29T11:11:20.873 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:11:20.873 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-03-29T11:11:20.873 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:11:20.873 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:20.873 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-03-29T11:11:20.873 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-29T11:11:20.873 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-03-29T11:11:20.873 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:20.873 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 171 k 2024-03-29T11:11:20.873 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 494 k 2024-03-29T11:11:20.873 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-03-29T11:11:21.108 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 732 kB/s | 171 kB 00:00 2024-03-29T11:11:21.109 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:21.109 INFO:teuthology.orchestra.run.smithi184.stdout:Total 727 kB/s | 171 kB 00:00 2024-03-29T11:11:21.109 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-03-29T11:11:21.117 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-03-29T11:11:21.117 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-03-29T11:11:21.173 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-03-29T11:11:21.173 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-03-29T11:11:21.326 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-03-29T11:11:21.431 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:21.828 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:22.116 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:22.116 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:22.116 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-03-29T11:11:22.116 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:11:22.116 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:22.116 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-29T11:11:22.345 DEBUG:teuthology.parallel:result is None 2024-03-29T11:11:22.345 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:11:22.491 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:11:22.517 INFO:teuthology.orchestra.run.smithi170.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:11:22.518 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:11:22.518 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:11:22.520 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:11:22.778 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:11:22.799 INFO:teuthology.orchestra.run.smithi184.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:11:22.800 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:11:22.800 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:11:22.802 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:11:22.802 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:11:22.802 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:11:22.835 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-29T11:11:22.835 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:11:22.868 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:11:22.868 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:11:22.868 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:11:22.906 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:11:22.979 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-29T11:11:22.979 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:11:23.007 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:11:23.074 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:11:23.074 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:11:23.075 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:11:23.104 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:11:23.173 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-29T11:11:23.173 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:11:23.200 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:11:23.267 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:11:23.267 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:11:23.267 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:11:23.295 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:11:23.363 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-29T11:11:23.364 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:11:23.390 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:11:23.457 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:11:23.536 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:11:23.536 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 777 /var/log/ceph 2024-03-29T11:11:23.539 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 777 /var/log/ceph 2024-03-29T11:11:23.566 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:11:23.567 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:11:23.609 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:11:23.635 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:11:23.635 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:11:23.677 DEBUG:teuthology.orchestra.run.smithi184:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:11:23.713 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:11:23.714 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:11:23.714 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622678', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622678', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi170.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK8KF8LxGDTr84GrjnvlzOw6qKh2eoa6jz9fKWQDjuMowPMVTuuGN5hLqxNs4JjbYvSy47O9KcB53fNM3ZmiQp8=', 'smithi184.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBChIWo4sRluXRqRfde3HW9mpN2k3+EawSGeT0X7df0O5eilEPv32w+Elg4a1z+ulVm2/6dGkwu0juAUsVblJt10='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:11:23.714 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:11:23.745 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:11:23.760 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:11:23.799 DEBUG:teuthology.orchestra.run.smithi184:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:11:23.827 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:11:23.827 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:11:23.880 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:11:23.881 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_1 2024-03-29T11:11:23.935 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:11:23.936 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:11:23.936 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T11:11:23.936 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:11:23.936 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:11:23.936 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-29 11:09:24.873714185 +0000 2024-03-29T11:11:23.936 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-29 11:06:19.517460135 +0000 2024-03-29T11:11:23.936 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-29 11:06:19.517460135 +0000 2024-03-29T11:11:23.936 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-29 11:06:19.517460135 +0000 2024-03-29T11:11:23.936 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:11:24.002 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-29T11:11:24.002 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-29T11:11:24.002 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 8.5932e-05 s, 6.0 MB/s 2024-03-29T11:11:24.003 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:11:24.059 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_2 2024-03-29T11:11:24.114 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:11:24.114 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:11:24.114 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T11:11:24.115 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:11:24.115 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:11:24.115 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-29 11:09:24.873714185 +0000 2024-03-29T11:11:24.115 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-29 11:06:19.845452543 +0000 2024-03-29T11:11:24.115 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-29 11:06:19.845452543 +0000 2024-03-29T11:11:24.115 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-29 11:06:19.845452543 +0000 2024-03-29T11:11:24.115 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:11:24.180 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-29T11:11:24.180 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-29T11:11:24.180 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 8.0928e-05 s, 6.3 MB/s 2024-03-29T11:11:24.181 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:11:24.237 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_3 2024-03-29T11:11:24.292 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:11:24.292 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:11:24.292 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T11:11:24.292 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:11:24.292 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:11:24.292 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-29 11:09:24.873714185 +0000 2024-03-29T11:11:24.292 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-29 11:06:20.168445066 +0000 2024-03-29T11:11:24.293 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-29 11:06:20.168445066 +0000 2024-03-29T11:11:24.293 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-29 11:06:20.168445066 +0000 2024-03-29T11:11:24.293 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:11:24.357 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-29T11:11:24.357 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-29T11:11:24.358 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 8.1525e-05 s, 6.3 MB/s 2024-03-29T11:11:24.358 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:11:24.414 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_4 2024-03-29T11:11:24.469 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:11:24.469 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:11:24.469 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-29T11:11:24.469 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:11:24.470 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:11:24.470 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-29 11:09:24.874714162 +0000 2024-03-29T11:11:24.470 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-29 11:06:20.513437079 +0000 2024-03-29T11:11:24.470 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-29 11:06:20.513437079 +0000 2024-03-29T11:11:24.470 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-29 11:06:20.513437079 +0000 2024-03-29T11:11:24.470 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:11:24.534 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-29T11:11:24.535 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-29T11:11:24.535 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.00010305 s, 5.0 MB/s 2024-03-29T11:11:24.536 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:11:24.592 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:24.592 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-29T11:11:24.592 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:11:24.611 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:11:24.611 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_1 2024-03-29T11:11:24.669 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:11:24.669 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:11:24.669 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T11:11:24.669 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:11:24.669 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:11:24.669 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-03-29 11:11:18.994902532 +0000 2024-03-29T11:11:24.669 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-03-29 11:06:22.159904064 +0000 2024-03-29T11:11:24.669 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-03-29 11:06:22.159904064 +0000 2024-03-29T11:11:24.669 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-03-29 11:06:22.159904064 +0000 2024-03-29T11:11:24.670 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:11:24.738 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-03-29T11:11:24.738 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-03-29T11:11:24.738 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000261944 s, 2.0 MB/s 2024-03-29T11:11:24.739 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:11:24.797 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_2 2024-03-29T11:11:24.854 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:11:24.854 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:11:24.854 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T11:11:24.854 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:11:24.854 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:11:24.854 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-03-29 11:11:18.994902532 +0000 2024-03-29T11:11:24.854 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-03-29 11:06:22.489896892 +0000 2024-03-29T11:11:24.854 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-03-29 11:06:22.489896892 +0000 2024-03-29T11:11:24.854 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-03-29 11:06:22.489896892 +0000 2024-03-29T11:11:24.855 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:11:24.922 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-03-29T11:11:24.922 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-03-29T11:11:24.922 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.00020472 s, 2.5 MB/s 2024-03-29T11:11:24.923 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:11:24.981 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_3 2024-03-29T11:11:25.037 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:11:25.037 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:11:25.037 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T11:11:25.038 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:11:25.038 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:11:25.038 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-03-29 11:11:18.995902510 +0000 2024-03-29T11:11:25.038 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-03-29 11:06:22.842889221 +0000 2024-03-29T11:11:25.038 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-03-29 11:06:22.842889221 +0000 2024-03-29T11:11:25.038 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-03-29 11:06:22.842889221 +0000 2024-03-29T11:11:25.038 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:11:25.105 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-03-29T11:11:25.105 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-03-29T11:11:25.105 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000231333 s, 2.2 MB/s 2024-03-29T11:11:25.106 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:11:25.163 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_4 2024-03-29T11:11:25.220 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:11:25.220 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:11:25.220 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T11:11:25.220 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:11:25.220 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:11:25.220 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-03-29 11:11:18.995902510 +0000 2024-03-29T11:11:25.220 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-03-29 11:06:23.182881832 +0000 2024-03-29T11:11:25.220 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-03-29 11:06:23.182881832 +0000 2024-03-29T11:11:25.221 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-03-29 11:06:23.182881832 +0000 2024-03-29T11:11:25.221 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:11:25.288 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-03-29T11:11:25.288 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-03-29T11:11:25.288 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.00018823 s, 2.7 MB/s 2024-03-29T11:11:25.289 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:11:25.345 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:25.346 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi184.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:11:25.346 INFO:tasks.ceph:Generating config... 2024-03-29T11:11:25.347 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:11:25.347 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:11:25.347 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:11:25.347 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:11:25.347 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:11:25.347 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:11:25.347 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:11:25.347 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:11:25.348 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:11:25.348 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:11:25.348 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:11:25.348 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:11:25.348 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:11:25.348 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:11:25.348 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:11:25.348 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:11:25.348 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:11:25.348 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:11:25.348 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:11:25.348 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:11:25.348 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:11:25.349 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:11:25.349 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:11:25.349 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T11:11:25.349 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:11:25.349 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:11:25.349 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:11:25.349 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:11:25.349 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:11:25.349 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:11:25.349 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:11:25.349 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:11:25.349 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:11:25.350 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:11:25.350 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:11:25.350 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:11:25.350 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:11:25.350 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:11:25.350 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:11:25.350 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:11:25.350 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:11:25.350 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:11:25.350 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:11:25.350 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:11:25.350 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:11:25.350 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:11:25.351 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:11:25.351 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:11:25.351 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:11:25.351 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:11:25.414 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:11:25.417 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:11:25.464 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:11:25.491 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.170:6789]'), ('mon.c', '[v1:172.21.15.170:6790]'), ('mon.b', '[v1:172.21.15.184:6789]')] 2024-03-29T11:11:25.492 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.170:6789],[v1:172.21.15.170:6790],[v1:172.21.15.184:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:11:25.493 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:11:25.493 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:11:25.547 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.170:6789]' --addv c '[v1:172.21.15.170:6790]' --addv b '[v1:172.21.15.184:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:25.633 INFO:teuthology.orchestra.run.smithi170.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:25.633 INFO:teuthology.orchestra.run.smithi170.stdout:monmaptool: generated fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:11:25.633 INFO:teuthology.orchestra.run.smithi170.stdout:setting min_mon_release = quincy 2024-03-29T11:11:25.633 INFO:teuthology.orchestra.run.smithi170.stdout:epoch 0 2024-03-29T11:11:25.633 INFO:teuthology.orchestra.run.smithi170.stdout:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:11:25.633 INFO:teuthology.orchestra.run.smithi170.stdout:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:11:25.633 INFO:teuthology.orchestra.run.smithi170.stdout:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:11:25.633 INFO:teuthology.orchestra.run.smithi170.stdout:min_mon_release 17 (quincy) 2024-03-29T11:11:25.633 INFO:teuthology.orchestra.run.smithi170.stdout:election_strategy: 1 2024-03-29T11:11:25.634 INFO:teuthology.orchestra.run.smithi170.stdout:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:11:25.634 INFO:teuthology.orchestra.run.smithi170.stdout:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:11:25.634 INFO:teuthology.orchestra.run.smithi170.stdout:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:11:25.634 INFO:teuthology.orchestra.run.smithi170.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:11:25.635 DEBUG:teuthology.orchestra.run.smithi170:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:11:25.650 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e9c3a847-ec94-40b5-8cc1-d1578b06cbfd... 2024-03-29T11:11:25.651 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:11:25.693 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:11:25.741 INFO:teuthology.orchestra.run.smithi170.stdout:[global] 2024-03-29T11:11:25.741 INFO:teuthology.orchestra.run.smithi170.stdout: chdir = "" 2024-03-29T11:11:25.741 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:11:25.741 INFO:teuthology.orchestra.run.smithi170.stdout: auth supported = cephx 2024-03-29T11:11:25.741 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.742 INFO:teuthology.orchestra.run.smithi170.stdout: filestore xattr use omap = true 2024-03-29T11:11:25.742 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.742 INFO:teuthology.orchestra.run.smithi170.stdout: mon clock drift allowed = 1.000 2024-03-29T11:11:25.742 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.742 INFO:teuthology.orchestra.run.smithi170.stdout: osd crush chooseleaf type = 0 2024-03-29T11:11:25.742 INFO:teuthology.orchestra.run.smithi170.stdout: auth debug = true 2024-03-29T11:11:25.742 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.742 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on old message = true 2024-03-29T11:11:25.742 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on bug = true 2024-03-29T11:11:25.742 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.742 INFO:teuthology.orchestra.run.smithi170.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:11:25.742 INFO:teuthology.orchestra.run.smithi170.stdout: mon pg warn max object skew = 0 2024-03-29T11:11:25.742 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.742 INFO:teuthology.orchestra.run.smithi170.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:11:25.742 INFO:teuthology.orchestra.run.smithi170.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:11:25.743 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.743 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default size = 2 2024-03-29T11:11:25.743 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.743 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow primary affinity = true 2024-03-29T11:11:25.743 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow pg remap = true 2024-03-29T11:11:25.743 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:11:25.743 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:11:25.743 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on no sortbitwise = false 2024-03-29T11:11:25.743 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:11:25.743 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on too few osds = false 2024-03-29T11:11:25.743 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:11:25.743 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:11:25.743 INFO:teuthology.orchestra.run.smithi170.stdout: mon_allow_pool_size_one = true 2024-03-29T11:11:25.743 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.744 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:11:25.744 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.744 INFO:teuthology.orchestra.run.smithi170.stdout: osd default data pool replay window = 5 2024-03-29T11:11:25.744 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.744 INFO:teuthology.orchestra.run.smithi170.stdout: mon allow pool delete = true 2024-03-29T11:11:25.744 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.744 INFO:teuthology.orchestra.run.smithi170.stdout: mon cluster log file level = debug 2024-03-29T11:11:25.744 INFO:teuthology.orchestra.run.smithi170.stdout: debug asserts on shutdown = true 2024-03-29T11:11:25.744 INFO:teuthology.orchestra.run.smithi170.stdout: mon health detail to clog = false 2024-03-29T11:11:25.744 INFO:teuthology.orchestra.run.smithi170.stdout: mon host = "[v1:172.21.15.170:6789],[v1:172.21.15.170:6790],[v1:172.21.15.184:6789]" 2024-03-29T11:11:25.744 INFO:teuthology.orchestra.run.smithi170.stdout: lockdep = True 2024-03-29T11:11:25.744 INFO:teuthology.orchestra.run.smithi170.stdout: mon client directed command retry = 5 2024-03-29T11:11:25.744 INFO:teuthology.orchestra.run.smithi170.stdout: mon election default strategy = 1 2024-03-29T11:11:25.744 INFO:teuthology.orchestra.run.smithi170.stdout: ms bind msgr2 = False 2024-03-29T11:11:25.744 INFO:teuthology.orchestra.run.smithi170.stdout: ms inject socket failures = 5000 2024-03-29T11:11:25.745 INFO:teuthology.orchestra.run.smithi170.stdout: ms type = async 2024-03-29T11:11:25.745 INFO:teuthology.orchestra.run.smithi170.stdout: fsid = e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:11:25.745 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.745 INFO:teuthology.orchestra.run.smithi170.stdout:[osd] 2024-03-29T11:11:25.745 INFO:teuthology.orchestra.run.smithi170.stdout: osd journal size = 100 2024-03-29T11:11:25.745 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.745 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub load threshold = 5.0 2024-03-29T11:11:25.745 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub max interval = 600 2024-03-29T11:11:25.745 INFO:teuthology.orchestra.run.smithi170.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:11:25.745 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.745 INFO:teuthology.orchestra.run.smithi170.stdout: osd recover clone overlap = true 2024-03-29T11:11:25.745 INFO:teuthology.orchestra.run.smithi170.stdout: osd recovery max chunk = 1048576 2024-03-29T11:11:25.745 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.745 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug shutdown = true 2024-03-29T11:11:25.745 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug op order = true 2024-03-29T11:11:25.746 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify stray on activate = true 2024-03-29T11:11:25.746 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug trim objects = true 2024-03-29T11:11:25.746 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.746 INFO:teuthology.orchestra.run.smithi170.stdout: osd open classes on start = true 2024-03-29T11:11:25.746 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug pg log writeout = true 2024-03-29T11:11:25.746 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.746 INFO:teuthology.orchestra.run.smithi170.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:11:25.746 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.746 INFO:teuthology.orchestra.run.smithi170.stdout: osd map max advance = 10 2024-03-29T11:11:25.746 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.746 INFO:teuthology.orchestra.run.smithi170.stdout: journal zero on create = true 2024-03-29T11:11:25.746 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.746 INFO:teuthology.orchestra.run.smithi170.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:11:25.747 INFO:teuthology.orchestra.run.smithi170.stdout: filestore apply finisher threads = 3 2024-03-29T11:11:25.747 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.747 INFO:teuthology.orchestra.run.smithi170.stdout: bdev debug aio = true 2024-03-29T11:11:25.747 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug misdirected ops = true 2024-03-29T11:11:25.747 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore block size = 96636764160 2024-03-29T11:11:25.747 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore compression algorithm = zstd 2024-03-29T11:11:25.747 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore compression mode = aggressive 2024-03-29T11:11:25.747 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore fsck on mount = True 2024-03-29T11:11:25.747 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore zero block detection = True 2024-03-29T11:11:25.747 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluefs = 20 2024-03-29T11:11:25.747 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluestore = 20 2024-03-29T11:11:25.747 INFO:teuthology.orchestra.run.smithi170.stdout: debug monc = 20 2024-03-29T11:11:25.747 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2024-03-29T11:11:25.747 INFO:teuthology.orchestra.run.smithi170.stdout: debug osd = 20 2024-03-29T11:11:25.747 INFO:teuthology.orchestra.run.smithi170.stdout: debug rocksdb = 10 2024-03-29T11:11:25.748 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:11:25.748 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd full ratio = 0.9 2024-03-29T11:11:25.748 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:11:25.748 INFO:teuthology.orchestra.run.smithi170.stdout: osd class default list = * 2024-03-29T11:11:25.748 INFO:teuthology.orchestra.run.smithi170.stdout: osd class load list = * 2024-03-29T11:11:25.748 INFO:teuthology.orchestra.run.smithi170.stdout: osd client watch timeout = 120 2024-03-29T11:11:25.748 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify cached snaps = True 2024-03-29T11:11:25.748 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify missing on start = True 2024-03-29T11:11:25.748 INFO:teuthology.orchestra.run.smithi170.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:11:25.748 INFO:teuthology.orchestra.run.smithi170.stdout: osd mclock override recovery settings = True 2024-03-29T11:11:25.748 INFO:teuthology.orchestra.run.smithi170.stdout: osd objectstore = bluestore 2024-03-29T11:11:25.748 INFO:teuthology.orchestra.run.smithi170.stdout: osd op queue = debug_random 2024-03-29T11:11:25.748 INFO:teuthology.orchestra.run.smithi170.stdout: osd op queue cut off = debug_random 2024-03-29T11:11:25.748 INFO:teuthology.orchestra.run.smithi170.stdout: osd shutdown pgref assert = True 2024-03-29T11:11:25.748 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.749 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr] 2024-03-29T11:11:25.749 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2024-03-29T11:11:25.749 INFO:teuthology.orchestra.run.smithi170.stdout: debug mgr = 20 2024-03-29T11:11:25.749 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2024-03-29T11:11:25.749 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2024-03-29T11:11:25.749 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:11:25.749 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:11:25.749 INFO:teuthology.orchestra.run.smithi170.stdout: mgr/telemetry/nag = false 2024-03-29T11:11:25.750 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.750 INFO:teuthology.orchestra.run.smithi170.stdout:[mon] 2024-03-29T11:11:25.750 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2024-03-29T11:11:25.750 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2024-03-29T11:11:25.750 INFO:teuthology.orchestra.run.smithi170.stdout: debug paxos = 20 2024-03-29T11:11:25.750 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2024-03-29T11:11:25.750 INFO:teuthology.orchestra.run.smithi170.stdout: mon data avail warn = 5 2024-03-29T11:11:25.750 INFO:teuthology.orchestra.run.smithi170.stdout: mon mgr mkfs grace = 240 2024-03-29T11:11:25.750 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:11:25.750 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd reporter subtree level = osd 2024-03-29T11:11:25.750 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd prime pg temp = true 2024-03-29T11:11:25.750 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:11:25.750 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.750 INFO:teuthology.orchestra.run.smithi170.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:11:25.751 INFO:teuthology.orchestra.run.smithi170.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:11:25.751 INFO:teuthology.orchestra.run.smithi170.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:11:25.751 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.751 INFO:teuthology.orchestra.run.smithi170.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:11:25.751 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:11:25.751 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:11:25.751 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.751 INFO:teuthology.orchestra.run.smithi170.stdout: # 1m isn't quite enough 2024-03-29T11:11:25.751 INFO:teuthology.orchestra.run.smithi170.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:11:25.751 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.751 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:11:25.751 INFO:teuthology.orchestra.run.smithi170.stdout: mon min osdmap epochs = 50 2024-03-29T11:11:25.751 INFO:teuthology.orchestra.run.smithi170.stdout: mon osdmap full prune interval = 2 2024-03-29T11:11:25.751 INFO:teuthology.orchestra.run.smithi170.stdout: mon osdmap full prune min = 15 2024-03-29T11:11:25.751 INFO:teuthology.orchestra.run.smithi170.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:11:25.752 INFO:teuthology.orchestra.run.smithi170.stdout: mon scrub interval = 300 2024-03-29T11:11:25.752 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on pool no app = False 2024-03-29T11:11:25.752 INFO:teuthology.orchestra.run.smithi170.stdout: paxos service trim min = 10 2024-03-29T11:11:25.752 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:11:25.752 INFO:teuthology.orchestra.run.smithi170.stdout:[client] 2024-03-29T11:11:25.752 INFO:teuthology.orchestra.run.smithi170.stdout: rgw cache enabled = true 2024-03-29T11:11:25.752 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable ops log = true 2024-03-29T11:11:25.752 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable usage log = true 2024-03-29T11:11:25.752 INFO:teuthology.orchestra.run.smithi170.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:11:25.752 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:11:25.752 INFO:teuthology.orchestra.run.smithi170.stdout: debug monc = 20 2024-03-29T11:11:25.752 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2024-03-29T11:11:25.752 INFO:teuthology.orchestra.run.smithi170.stdout: debug objecter = 20 2024-03-29T11:11:25.752 INFO:teuthology.orchestra.run.smithi170.stdout: debug rados = 20 2024-03-29T11:11:25.752 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.a] 2024-03-29T11:11:25.753 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.c] 2024-03-29T11:11:25.753 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.b] 2024-03-29T11:11:25.753 INFO:teuthology.orchestra.run.smithi184.stdout:[global] 2024-03-29T11:11:25.753 INFO:teuthology.orchestra.run.smithi184.stdout: chdir = "" 2024-03-29T11:11:25.753 INFO:teuthology.orchestra.run.smithi184.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:11:25.753 INFO:teuthology.orchestra.run.smithi184.stdout: auth supported = cephx 2024-03-29T11:11:25.754 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.754 INFO:teuthology.orchestra.run.smithi184.stdout: filestore xattr use omap = true 2024-03-29T11:11:25.754 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.754 INFO:teuthology.orchestra.run.smithi184.stdout: mon clock drift allowed = 1.000 2024-03-29T11:11:25.754 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.754 INFO:teuthology.orchestra.run.smithi184.stdout: osd crush chooseleaf type = 0 2024-03-29T11:11:25.754 INFO:teuthology.orchestra.run.smithi184.stdout: auth debug = true 2024-03-29T11:11:25.754 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.754 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on old message = true 2024-03-29T11:11:25.754 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on bug = true 2024-03-29T11:11:25.754 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.754 INFO:teuthology.orchestra.run.smithi184.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:11:25.754 INFO:teuthology.orchestra.run.smithi184.stdout: mon pg warn max object skew = 0 2024-03-29T11:11:25.755 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.755 INFO:teuthology.orchestra.run.smithi184.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:11:25.755 INFO:teuthology.orchestra.run.smithi184.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:11:25.755 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.755 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default size = 2 2024-03-29T11:11:25.755 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.755 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow primary affinity = true 2024-03-29T11:11:25.755 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow pg remap = true 2024-03-29T11:11:25.755 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:11:25.755 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:11:25.755 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on no sortbitwise = false 2024-03-29T11:11:25.755 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:11:25.755 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on too few osds = false 2024-03-29T11:11:25.755 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:11:25.756 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:11:25.756 INFO:teuthology.orchestra.run.smithi184.stdout: mon_allow_pool_size_one = true 2024-03-29T11:11:25.756 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.756 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:11:25.756 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.756 INFO:teuthology.orchestra.run.smithi184.stdout: osd default data pool replay window = 5 2024-03-29T11:11:25.756 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.756 INFO:teuthology.orchestra.run.smithi184.stdout: mon allow pool delete = true 2024-03-29T11:11:25.756 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.756 INFO:teuthology.orchestra.run.smithi184.stdout: mon cluster log file level = debug 2024-03-29T11:11:25.756 INFO:teuthology.orchestra.run.smithi184.stdout: debug asserts on shutdown = true 2024-03-29T11:11:25.756 INFO:teuthology.orchestra.run.smithi184.stdout: mon health detail to clog = false 2024-03-29T11:11:25.756 INFO:teuthology.orchestra.run.smithi184.stdout: mon host = "[v1:172.21.15.170:6789],[v1:172.21.15.170:6790],[v1:172.21.15.184:6789]" 2024-03-29T11:11:25.756 INFO:teuthology.orchestra.run.smithi184.stdout: lockdep = True 2024-03-29T11:11:25.757 INFO:teuthology.orchestra.run.smithi184.stdout: mon client directed command retry = 5 2024-03-29T11:11:25.757 INFO:teuthology.orchestra.run.smithi184.stdout: mon election default strategy = 1 2024-03-29T11:11:25.757 INFO:teuthology.orchestra.run.smithi184.stdout: ms bind msgr2 = False 2024-03-29T11:11:25.757 INFO:teuthology.orchestra.run.smithi184.stdout: ms inject socket failures = 5000 2024-03-29T11:11:25.757 INFO:teuthology.orchestra.run.smithi184.stdout: ms type = async 2024-03-29T11:11:25.757 INFO:teuthology.orchestra.run.smithi184.stdout: fsid = e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:11:25.757 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.757 INFO:teuthology.orchestra.run.smithi184.stdout:[osd] 2024-03-29T11:11:25.757 INFO:teuthology.orchestra.run.smithi184.stdout: osd journal size = 100 2024-03-29T11:11:25.757 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.757 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub load threshold = 5.0 2024-03-29T11:11:25.757 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub max interval = 600 2024-03-29T11:11:25.757 INFO:teuthology.orchestra.run.smithi184.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:11:25.757 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.757 INFO:teuthology.orchestra.run.smithi184.stdout: osd recover clone overlap = true 2024-03-29T11:11:25.758 INFO:teuthology.orchestra.run.smithi184.stdout: osd recovery max chunk = 1048576 2024-03-29T11:11:25.758 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.758 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug shutdown = true 2024-03-29T11:11:25.758 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug op order = true 2024-03-29T11:11:25.758 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug verify stray on activate = true 2024-03-29T11:11:25.758 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug trim objects = true 2024-03-29T11:11:25.758 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.758 INFO:teuthology.orchestra.run.smithi184.stdout: osd open classes on start = true 2024-03-29T11:11:25.758 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug pg log writeout = true 2024-03-29T11:11:25.758 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.758 INFO:teuthology.orchestra.run.smithi184.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:11:25.758 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.758 INFO:teuthology.orchestra.run.smithi184.stdout: osd map max advance = 10 2024-03-29T11:11:25.759 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.759 INFO:teuthology.orchestra.run.smithi184.stdout: journal zero on create = true 2024-03-29T11:11:25.759 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.759 INFO:teuthology.orchestra.run.smithi184.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:11:25.759 INFO:teuthology.orchestra.run.smithi184.stdout: filestore apply finisher threads = 3 2024-03-29T11:11:25.759 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.759 INFO:teuthology.orchestra.run.smithi184.stdout: bdev debug aio = true 2024-03-29T11:11:25.759 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug misdirected ops = true 2024-03-29T11:11:25.759 INFO:teuthology.orchestra.run.smithi184.stdout: bluestore block size = 96636764160 2024-03-29T11:11:25.759 INFO:teuthology.orchestra.run.smithi184.stdout: bluestore compression algorithm = zstd 2024-03-29T11:11:25.759 INFO:teuthology.orchestra.run.smithi184.stdout: bluestore compression mode = aggressive 2024-03-29T11:11:25.759 INFO:teuthology.orchestra.run.smithi184.stdout: bluestore fsck on mount = True 2024-03-29T11:11:25.759 INFO:teuthology.orchestra.run.smithi184.stdout: bluestore zero block detection = True 2024-03-29T11:11:25.760 INFO:teuthology.orchestra.run.smithi184.stdout: debug bluefs = 20 2024-03-29T11:11:25.760 INFO:teuthology.orchestra.run.smithi184.stdout: debug bluestore = 20 2024-03-29T11:11:25.760 INFO:teuthology.orchestra.run.smithi184.stdout: debug monc = 20 2024-03-29T11:11:25.760 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2024-03-29T11:11:25.760 INFO:teuthology.orchestra.run.smithi184.stdout: debug osd = 20 2024-03-29T11:11:25.760 INFO:teuthology.orchestra.run.smithi184.stdout: debug rocksdb = 10 2024-03-29T11:11:25.760 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:11:25.760 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd full ratio = 0.9 2024-03-29T11:11:25.760 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:11:25.760 INFO:teuthology.orchestra.run.smithi184.stdout: osd class default list = * 2024-03-29T11:11:25.760 INFO:teuthology.orchestra.run.smithi184.stdout: osd class load list = * 2024-03-29T11:11:25.760 INFO:teuthology.orchestra.run.smithi184.stdout: osd client watch timeout = 120 2024-03-29T11:11:25.760 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug verify cached snaps = True 2024-03-29T11:11:25.760 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug verify missing on start = True 2024-03-29T11:11:25.761 INFO:teuthology.orchestra.run.smithi184.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:11:25.761 INFO:teuthology.orchestra.run.smithi184.stdout: osd mclock override recovery settings = True 2024-03-29T11:11:25.761 INFO:teuthology.orchestra.run.smithi184.stdout: osd objectstore = bluestore 2024-03-29T11:11:25.761 INFO:teuthology.orchestra.run.smithi184.stdout: osd op queue = debug_random 2024-03-29T11:11:25.761 INFO:teuthology.orchestra.run.smithi184.stdout: osd op queue cut off = debug_random 2024-03-29T11:11:25.761 INFO:teuthology.orchestra.run.smithi184.stdout: osd shutdown pgref assert = True 2024-03-29T11:11:25.761 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.761 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr] 2024-03-29T11:11:25.761 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2024-03-29T11:11:25.761 INFO:teuthology.orchestra.run.smithi184.stdout: debug mgr = 20 2024-03-29T11:11:25.761 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2024-03-29T11:11:25.761 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2024-03-29T11:11:25.761 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:11:25.761 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:11:25.761 INFO:teuthology.orchestra.run.smithi184.stdout: mgr/telemetry/nag = false 2024-03-29T11:11:25.762 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.762 INFO:teuthology.orchestra.run.smithi184.stdout:[mon] 2024-03-29T11:11:25.762 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2024-03-29T11:11:25.762 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2024-03-29T11:11:25.762 INFO:teuthology.orchestra.run.smithi184.stdout: debug paxos = 20 2024-03-29T11:11:25.762 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2024-03-29T11:11:25.762 INFO:teuthology.orchestra.run.smithi184.stdout: mon data avail warn = 5 2024-03-29T11:11:25.762 INFO:teuthology.orchestra.run.smithi184.stdout: mon mgr mkfs grace = 240 2024-03-29T11:11:25.762 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:11:25.762 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd reporter subtree level = osd 2024-03-29T11:11:25.762 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd prime pg temp = true 2024-03-29T11:11:25.762 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:11:25.762 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.762 INFO:teuthology.orchestra.run.smithi184.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:11:25.762 INFO:teuthology.orchestra.run.smithi184.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:11:25.763 INFO:teuthology.orchestra.run.smithi184.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:11:25.763 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.763 INFO:teuthology.orchestra.run.smithi184.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:11:25.763 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:11:25.763 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:11:25.763 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.763 INFO:teuthology.orchestra.run.smithi184.stdout: # 1m isn't quite enough 2024-03-29T11:11:25.763 INFO:teuthology.orchestra.run.smithi184.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:11:25.763 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.763 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:11:25.763 INFO:teuthology.orchestra.run.smithi184.stdout: mon min osdmap epochs = 50 2024-03-29T11:11:25.763 INFO:teuthology.orchestra.run.smithi184.stdout: mon osdmap full prune interval = 2 2024-03-29T11:11:25.763 INFO:teuthology.orchestra.run.smithi184.stdout: mon osdmap full prune min = 15 2024-03-29T11:11:25.763 INFO:teuthology.orchestra.run.smithi184.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:11:25.763 INFO:teuthology.orchestra.run.smithi184.stdout: mon scrub interval = 300 2024-03-29T11:11:25.764 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on pool no app = False 2024-03-29T11:11:25.764 INFO:teuthology.orchestra.run.smithi184.stdout: paxos service trim min = 10 2024-03-29T11:11:25.764 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-29T11:11:25.764 INFO:teuthology.orchestra.run.smithi184.stdout:[client] 2024-03-29T11:11:25.764 INFO:teuthology.orchestra.run.smithi184.stdout: rgw cache enabled = true 2024-03-29T11:11:25.764 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable ops log = true 2024-03-29T11:11:25.764 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable usage log = true 2024-03-29T11:11:25.764 INFO:teuthology.orchestra.run.smithi184.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:11:25.764 INFO:teuthology.orchestra.run.smithi184.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:11:25.764 INFO:teuthology.orchestra.run.smithi184.stdout: debug monc = 20 2024-03-29T11:11:25.764 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2024-03-29T11:11:25.764 INFO:teuthology.orchestra.run.smithi184.stdout: debug objecter = 20 2024-03-29T11:11:25.764 INFO:teuthology.orchestra.run.smithi184.stdout: debug rados = 20 2024-03-29T11:11:25.764 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.a] 2024-03-29T11:11:25.764 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.c] 2024-03-29T11:11:25.765 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.b] 2024-03-29T11:11:25.766 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:11:25.766 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:11:25.822 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:11:25.822 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:11:25.822 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:11:25.837 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:11:25.837 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:11:25.892 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi170.front.sepia.ceph.com 2024-03-29T11:11:25.893 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:11:25.893 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:11:25.893 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:11:25.972 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:11:25.972 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:26.027 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi184.front.sepia.ceph.com 2024-03-29T11:11:26.027 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-29T11:11:26.027 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:11:26.027 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:11:26.071 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-29T11:11:26.072 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:26.128 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:11:26.128 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:11:26.128 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:11:26.183 INFO:teuthology.orchestra.run.smithi170.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:11:26.185 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:11:26.260 INFO:teuthology.orchestra.run.smithi184.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:11:26.263 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:11:26.263 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:11:26.263 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:11:26.306 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:11:26.321 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:11:26.366 INFO:teuthology.orchestra.run.smithi184.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:11:26.382 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:11:26.383 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi184.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:11:26.383 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:11:26.410 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:26.410 INFO:tasks.ceph:role: osd.0 2024-03-29T11:11:26.410 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com 2024-03-29T11:11:26.410 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:11:26.509 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2024-03-29T11:11:26.511 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2024-03-29T11:11:26.511 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:11:26.582 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:11:26.654 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:11:26.722 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:26.722 INFO:tasks.ceph:role: osd.1 2024-03-29T11:11:26.722 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com 2024-03-29T11:11:26.722 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:11:26.788 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:11:26.788 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:11:26.788 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:11:26.788 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:11:26.788 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:11:26.789 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:11:26.789 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:11:26.789 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:11:26.789 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:11:26.789 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:11:26.821 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2024-03-29T11:11:26.822 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2024-03-29T11:11:26.822 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:11:26.892 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:11:26.967 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:11:27.034 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:27.034 INFO:tasks.ceph:role: osd.2 2024-03-29T11:11:27.034 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi170.front.sepia.ceph.com 2024-03-29T11:11:27.035 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:11:27.101 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:11:27.101 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:11:27.101 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:11:27.101 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:11:27.101 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:11:27.101 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:11:27.101 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:11:27.101 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:11:27.101 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:11:27.101 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:11:27.134 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2024-03-29T11:11:27.136 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2024-03-29T11:11:27.136 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:11:27.212 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:11:27.285 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:11:27.352 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:27.352 INFO:tasks.ceph:role: osd.3 2024-03-29T11:11:27.353 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi170.front.sepia.ceph.com 2024-03-29T11:11:27.353 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:11:27.419 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:11:27.419 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:11:27.419 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:11:27.419 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:11:27.419 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:11:27.419 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:11:27.420 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:11:27.420 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:11:27.420 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:11:27.420 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:11:27.451 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2024-03-29T11:11:27.453 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2024-03-29T11:11:27.453 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:11:27.523 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:11:27.594 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:27.689 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:27.710+0000 7ff6905ac640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:11:27.689 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:27.710+0000 7ff6905ac640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:11:27.689 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:27.710+0000 7ff6905ac640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:11:27.689 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:27.710+0000 7ff6905ac640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:11:27.689 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:27.710+0000 7ff6905ac640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:11:27.690 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:27.710+0000 7ff6905ac640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:11:30.537 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:11:30.568 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:30.659 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:30.680+0000 7f7efdddd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:11:30.659 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:30.680+0000 7f7efdddd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:11:30.659 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:30.680+0000 7f7efdddd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:11:30.660 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:30.680+0000 7f7efdddd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:11:30.660 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:30.680+0000 7f7efdddd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:11:30.660 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:30.681+0000 7f7efdddd640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:11:33.512 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:11:33.543 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:33.635 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:33.656+0000 7f0ef1a11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:11:33.636 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:33.657+0000 7f0ef1a11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:11:33.636 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:33.657+0000 7f0ef1a11640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:11:33.636 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:33.657+0000 7f0ef1a11640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:11:33.636 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:33.657+0000 7f0ef1a11640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:11:33.636 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:33.657+0000 7f0ef1a11640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:11:36.490 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:11:36.521 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:36.612 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:36.633+0000 7f925ce62640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:11:36.612 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:36.633+0000 7f925ce62640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:11:36.612 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:36.633+0000 7f925ce62640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:11:36.612 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:36.633+0000 7f925ce62640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:11:36.612 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:36.633+0000 7f925ce62640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:11:36.613 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:36.634+0000 7f925ce62640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:11:39.465 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:11:39.497 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:11:39.530 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:39.530 INFO:tasks.ceph:role: osd.4 2024-03-29T11:11:39.531 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi184.front.sepia.ceph.com 2024-03-29T11:11:39.531 DEBUG:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:11:39.609 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:11:39.609 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:11:39.609 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:11:39.609 INFO:teuthology.orchestra.run.smithi184.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:11:39.609 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:11:39.609 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:11:39.609 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:11:39.609 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:11:39.609 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:11:39.609 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:11:40.132 INFO:teuthology.orchestra.run.smithi184.stdout:Discarding blocks...Done. 2024-03-29T11:11:40.134 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2024-03-29T11:11:40.134 DEBUG:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:11:40.289 DEBUG:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:11:40.325 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:11:40.395 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:40.395 INFO:tasks.ceph:role: osd.5 2024-03-29T11:11:40.395 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi184.front.sepia.ceph.com 2024-03-29T11:11:40.395 DEBUG:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:11:40.471 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:11:40.471 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:11:40.471 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:11:40.471 INFO:teuthology.orchestra.run.smithi184.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:11:40.471 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:11:40.471 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:11:40.471 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:11:40.472 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:11:40.472 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:11:40.472 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:11:40.916 INFO:teuthology.orchestra.run.smithi184.stdout:Discarding blocks...Done. 2024-03-29T11:11:40.918 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2024-03-29T11:11:40.918 DEBUG:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:11:41.014 DEBUG:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:11:41.045 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:11:41.112 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:41.112 INFO:tasks.ceph:role: osd.6 2024-03-29T11:11:41.112 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi184.front.sepia.ceph.com 2024-03-29T11:11:41.113 DEBUG:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:11:41.187 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:11:41.187 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:11:41.187 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:11:41.187 INFO:teuthology.orchestra.run.smithi184.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:11:41.187 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:11:41.187 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:11:41.188 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:11:41.188 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:11:41.188 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:11:41.188 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:11:41.672 INFO:teuthology.orchestra.run.smithi184.stdout:Discarding blocks...Done. 2024-03-29T11:11:41.675 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2024-03-29T11:11:41.675 DEBUG:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:11:41.824 DEBUG:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:11:41.857 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:11:41.925 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:41.925 INFO:tasks.ceph:role: osd.7 2024-03-29T11:11:41.925 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi184.front.sepia.ceph.com 2024-03-29T11:11:41.926 DEBUG:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:11:41.999 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:11:41.999 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:11:41.999 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:11:41.999 INFO:teuthology.orchestra.run.smithi184.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:11:42.000 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:11:42.000 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:11:42.000 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:11:42.000 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:11:42.000 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:11:42.000 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:11:42.506 INFO:teuthology.orchestra.run.smithi184.stdout:Discarding blocks...Done. 2024-03-29T11:11:42.508 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2024-03-29T11:11:42.508 DEBUG:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:11:42.659 DEBUG:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:11:42.692 DEBUG:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:42.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:42.827+0000 7f2e7301e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:11:42.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:42.828+0000 7f2e7301e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:11:42.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:42.828+0000 7f2e7301e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:11:42.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:42.828+0000 7f2e7301e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:11:42.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:42.828+0000 7f2e7301e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:11:42.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:42.828+0000 7f2e7301e640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:11:45.661 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:11:45.694 DEBUG:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:45.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:45.808+0000 7fc16cea2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:11:45.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:45.808+0000 7fc16cea2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:11:45.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:45.808+0000 7fc16cea2640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:11:45.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:45.808+0000 7fc16cea2640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:11:45.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:45.808+0000 7fc16cea2640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:11:45.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:45.809+0000 7fc16cea2640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:11:48.634 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:11:48.667 DEBUG:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:48.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:48.783+0000 7fbf377ac640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:11:48.762 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:48.783+0000 7fbf377ac640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:11:48.762 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:48.783+0000 7fbf377ac640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:11:48.762 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:48.783+0000 7fbf377ac640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:11:48.762 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:48.783+0000 7fbf377ac640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:11:48.762 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:48.784+0000 7fbf377ac640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:11:51.635 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:11:51.666 DEBUG:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:51.755 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:51.777+0000 7fdf3fe11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:11:51.755 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:51.777+0000 7fdf3fe11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:11:51.755 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:51.778+0000 7fdf3fe11640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:11:51.755 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:51.778+0000 7fdf3fe11640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:11:51.755 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:51.778+0000 7fdf3fe11640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:11:51.755 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:11:51.778+0000 7fdf3fe11640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:11:54.616 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:11:54.646 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:11:54.646 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:11:54.646 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:11:54.679 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:11:54.679 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:11:54.750 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:11:54.750 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:11:54.821 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:11:54.821 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:11:54.891 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:11:54.891 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:11:54.959 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-29T11:11:54.959 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:11:54.991 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-29T11:11:54.991 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:11:55.062 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-29T11:11:55.062 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:11:55.131 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-29T11:11:55.131 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:11:55.200 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-29T11:11:55.200 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:11:55.268 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:11:55.269 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:11:55.287 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-29T11:11:55.287 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:11:55.326 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:11:55.326 DEBUG:teuthology.orchestra.run.smithi170:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:11:55.329 DEBUG:teuthology.orchestra.run.smithi184:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:11:55.369 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr.y] 2024-03-29T11:11:55.369 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDeoQZmwxRLDBAAFjTHHdwV1Ynz2XvosXOLUQ== 2024-03-29T11:11:55.369 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.0] 2024-03-29T11:11:55.369 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDfoQZmKpxuKhAAb3Qu2rowp6p6BNLCod9vrQ== 2024-03-29T11:11:55.369 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.1] 2024-03-29T11:11:55.369 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDioQZmwkaoKBAAy8Br0YbDrUJj1k32kw5Esg== 2024-03-29T11:11:55.370 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.2] 2024-03-29T11:11:55.370 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDloQZmsHM9JxAA8erlNtEQFySbBJawl/zRjg== 2024-03-29T11:11:55.370 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.3] 2024-03-29T11:11:55.370 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDooQZmyWzZJRAAkxpd4quFjApfQmYC1UlS+w== 2024-03-29T11:11:55.370 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr.x] 2024-03-29T11:11:55.370 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDeoQZmeNroEBAA211OTgSXW7ym2jV759JoEQ== 2024-03-29T11:11:55.370 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.4] 2024-03-29T11:11:55.370 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDuoQZmV6RtMRAAymTzIF1opouyKVVLUTlbfA== 2024-03-29T11:11:55.370 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.5] 2024-03-29T11:11:55.370 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDxoQZmAkpGMBAAzjCIOnySoPdUbhFZtMNPlQ== 2024-03-29T11:11:55.370 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.6] 2024-03-29T11:11:55.370 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQD0oQZm3kq4LhAAERxHZ0Br6Xsceb3WPMfkkA== 2024-03-29T11:11:55.371 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.7] 2024-03-29T11:11:55.371 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQD3oQZmp29fLhAAFJPpvFFUD+31lFTKjp8Q7A== 2024-03-29T11:11:55.371 INFO:teuthology.orchestra.run.smithi170.stdout:[client.0] 2024-03-29T11:11:55.371 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDeoQZmhSCiExAAJ7VJI4BEwVQCbkFFKiTvMA== 2024-03-29T11:11:55.371 INFO:teuthology.orchestra.run.smithi170.stdout:[client.1] 2024-03-29T11:11:55.371 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDeoQZmFG4wFxAA4PG3VD+rpXhfM/Z3oEaF4w== 2024-03-29T11:11:55.392 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr.y] 2024-03-29T11:11:55.392 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDeoQZmwxRLDBAAFjTHHdwV1Ynz2XvosXOLUQ== 2024-03-29T11:11:55.392 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.0] 2024-03-29T11:11:55.392 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDfoQZmKpxuKhAAb3Qu2rowp6p6BNLCod9vrQ== 2024-03-29T11:11:55.392 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.1] 2024-03-29T11:11:55.393 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDioQZmwkaoKBAAy8Br0YbDrUJj1k32kw5Esg== 2024-03-29T11:11:55.393 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.2] 2024-03-29T11:11:55.393 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDloQZmsHM9JxAA8erlNtEQFySbBJawl/zRjg== 2024-03-29T11:11:55.393 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.3] 2024-03-29T11:11:55.393 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDooQZmyWzZJRAAkxpd4quFjApfQmYC1UlS+w== 2024-03-29T11:11:55.393 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr.x] 2024-03-29T11:11:55.393 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDeoQZmeNroEBAA211OTgSXW7ym2jV759JoEQ== 2024-03-29T11:11:55.393 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.4] 2024-03-29T11:11:55.393 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDuoQZmV6RtMRAAymTzIF1opouyKVVLUTlbfA== 2024-03-29T11:11:55.393 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.5] 2024-03-29T11:11:55.393 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDxoQZmAkpGMBAAzjCIOnySoPdUbhFZtMNPlQ== 2024-03-29T11:11:55.393 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.6] 2024-03-29T11:11:55.393 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQD0oQZm3kq4LhAAERxHZ0Br6Xsceb3WPMfkkA== 2024-03-29T11:11:55.393 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.7] 2024-03-29T11:11:55.394 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQD3oQZmp29fLhAAFJPpvFFUD+31lFTKjp8Q7A== 2024-03-29T11:11:55.394 INFO:teuthology.orchestra.run.smithi184.stdout:[client.0] 2024-03-29T11:11:55.394 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDeoQZmhSCiExAAJ7VJI4BEwVQCbkFFKiTvMA== 2024-03-29T11:11:55.394 INFO:teuthology.orchestra.run.smithi184.stdout:[client.1] 2024-03-29T11:11:55.394 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDeoQZmFG4wFxAA4PG3VD+rpXhfM/Z3oEaF4w== 2024-03-29T11:11:55.395 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:11:55.415 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:11:55.488 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:55.490 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:55.543 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:55.545 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:55.597 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:55.599 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:55.650 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:55.654 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:55.705 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:11:55.707 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:11:55.759 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:55.761 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:55.813 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:55.815 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:55.866 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:55.868 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:55.920 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:55.922 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:55.974 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:11:55.976 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:11:56.032 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:11:56.034 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:11:56.089 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:11:56.089 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:11:56.116 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:11:56.212 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:11:56.242 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:11:56.309 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:11:56.406 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:11:56.433 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:11:56.461 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:11:56.558 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:11:56.585 DEBUG:teuthology.orchestra.run.smithi170:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:56.587 DEBUG:teuthology.orchestra.run.smithi184:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:56.640 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:11:56.641 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:11:56.641 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:11:56.644 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:11:56.644 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:11:56.644 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:11:56.646 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:11:56.646 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:11:56.646 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:11:56.682 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:11:56.683 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:11:56.683 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:11:56.683 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:11:56.685 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:11:56.685 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:11:56.685 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:11:56.687 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:11:56.687 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:11:56.687 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:11:56.790 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:56.812+0000 7f558ca51640 10 monclient: get_monmap_and_config 2024-03-29T11:11:56.790 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:56.812+0000 7f558ca51640 10 monclient: build_initial_monmap 2024-03-29T11:11:56.790 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:56.812+0000 7f558ca51640 10 monclient: monmap: 2024-03-29T11:11:56.790 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2024-03-29T11:11:56.790 INFO:teuthology.orchestra.run.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:11:56.790 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2024-03-29T11:11:56.812688+0000 2024-03-29T11:11:56.791 INFO:teuthology.orchestra.run.smithi170.stderr:created 2024-03-29T11:11:56.812688+0000 2024-03-29T11:11:56.791 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:56.791 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2024-03-29T11:11:56.791 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:11:56.791 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:11:56.791 INFO:teuthology.orchestra.run.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:11:56.791 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-29T11:11:56.791 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:56.812+0000 7f558ca51640 1 Processor -- start 2024-03-29T11:11:56.791 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:56.812+0000 7f558ca51640 1 -- start start 2024-03-29T11:11:56.791 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:56.812+0000 7f558ca51640 10 monclient: init 2024-03-29T11:11:56.793 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:56.813+0000 7f558ca51640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:56.793 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:56.813+0000 7f558ca51640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:11:56.793 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:56.813+0000 7f558ca51640 10 monclient(hunting): picked mon.noname-b con 0x7f55880b8240 addr v1:172.21.15.170:6790/0 2024-03-29T11:11:56.793 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:56.813+0000 7f558ca51640 10 monclient(hunting): picked mon.noname-a con 0x7f55880b1f60 addr v1:172.21.15.170:6789/0 2024-03-29T11:11:56.793 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:56.813+0000 7f558ca51640 10 monclient(hunting): picked mon.noname-c con 0x7f558809c8c0 addr v1:172.21.15.184:6789/0 2024-03-29T11:11:56.793 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:56.813+0000 7f558ca51640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558809bd80 con 0x7f55880b1f60 2024-03-29T11:11:56.793 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:56.813+0000 7f558ca51640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558809b960 con 0x7f558809c8c0 2024-03-29T11:11:56.793 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:56.813+0000 7f558ca51640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55880b2ec0 con 0x7f55880b8240 2024-03-29T11:11:56.794 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:56.813+0000 7f558ca51640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:56.794 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:56.813+0000 7f558ca51640 10 monclient(hunting): authenticate will time out at 898.124680s 2024-03-29T11:11:56.794 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:56.813+0000 7f558659c640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f55880b8240 0x7f558813f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:43150/0 (socket says 172.21.15.170:43150) 2024-03-29T11:11:56.794 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:56.813+0000 7f5585d9b640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f55880b1f60 0x7f558809c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:33918/0 (socket says 172.21.15.170:33918) 2024-03-29T11:11:56.794 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:56.813+0000 7f558659c640 1 -- 172.21.15.170:0/4223247859 learned_addr learned my addr 172.21.15.170:0/4223247859 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:11:59.791 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:59.813+0000 7f5577fff640 10 monclient(hunting): tick 2024-03-29T11:11:59.791 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:59.813+0000 7f5577fff640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:11:59.791 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:59.813+0000 7f5577fff640 1 monclient(hunting): continuing hunt 2024-03-29T11:11:59.791 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:59.813+0000 7f5577fff640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:11:59.791 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:59.813+0000 7f5577fff640 1 -- 172.21.15.170:0/4223247859 >> v1:172.21.15.170:6790/0 conn(0x7f55880b8240 legacy=0x7f558813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:59.792 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:59.813+0000 7f5577fff640 1 -- 172.21.15.170:0/4223247859 >> v1:172.21.15.184:6789/0 conn(0x7f558809c8c0 legacy=0x7f558814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:59.792 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:59.813+0000 7f5577fff640 1 -- 172.21.15.170:0/4223247859 >> v1:172.21.15.170:6789/0 conn(0x7f55880b1f60 legacy=0x7f558809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:59.792 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:59.813+0000 7f5577fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:11:59.792 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:59.813+0000 7f5577fff640 10 monclient(hunting): picked mon.noname-b con 0x7f55680036e0 addr v1:172.21.15.170:6790/0 2024-03-29T11:11:59.792 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:59.813+0000 7f5577fff640 10 monclient(hunting): picked mon.noname-a con 0x7f55680072d0 addr v1:172.21.15.170:6789/0 2024-03-29T11:11:59.792 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:59.813+0000 7f5577fff640 10 monclient(hunting): picked mon.noname-c con 0x7f558809c8c0 addr v1:172.21.15.184:6789/0 2024-03-29T11:11:59.792 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:59.813+0000 7f5577fff640 1 -- 172.21.15.170:0/4223247859 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558813fb60 con 0x7f55680072d0 2024-03-29T11:11:59.792 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:59.813+0000 7f5577fff640 1 -- 172.21.15.170:0/4223247859 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556800f8d0 con 0x7f558809c8c0 2024-03-29T11:11:59.792 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:59.813+0000 7f5577fff640 1 -- 172.21.15.170:0/4223247859 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5568010b10 con 0x7f55680036e0 2024-03-29T11:11:59.792 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:11:59.813+0000 7f5577fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:01.963 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.986+0000 7f558559a640 1 -- 172.21.15.170:0/4223247859 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919417471 0 0) 0x7f558813fb60 con 0x7f55680072d0 2024-03-29T11:12:01.964 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.986+0000 7f558559a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:01.964 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.986+0000 7f558559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:01.964 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.986+0000 7f558559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:01.964 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.986+0000 7f558559a640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:12:01.964 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.986+0000 7f558559a640 1 -- 172.21.15.170:0/4223247859 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5570003520 con 0x7f55680072d0 2024-03-29T11:12:01.964 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.986+0000 7f558559a640 1 -- 172.21.15.170:0/4223247859 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839971702 0 0) 0x7f5568010b10 con 0x7f55680036e0 2024-03-29T11:12:01.964 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.986+0000 7f558559a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:01.964 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.986+0000 7f558559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:01.964 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.986+0000 7f558559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:01.964 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.986+0000 7f558559a640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:12:01.964 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.986+0000 7f558559a640 1 -- 172.21.15.170:0/4223247859 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f558813fb60 con 0x7f55680036e0 2024-03-29T11:12:01.966 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 1 -- 172.21.15.170:0/4223247859 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1489699617 0 0) 0x7f558813fb60 con 0x7f55680036e0 2024-03-29T11:12:01.966 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 1 -- 172.21.15.170:0/4223247859 >> v1:172.21.15.184:6789/0 conn(0x7f558809c8c0 legacy=0x7f556800d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:01.967 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 1 -- 172.21.15.170:0/4223247859 >> v1:172.21.15.170:6789/0 conn(0x7f55680072d0 legacy=0x7f5568009720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:01.967 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 10 monclient: _finish_hunting 0 2024-03-29T11:12:01.967 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 1 monclient: found mon.noname-b 2024-03-29T11:12:01.967 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:01.967 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:12:01.967 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 1 -- 172.21.15.170:0/4223247859 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5568011d70 con 0x7f55680036e0 2024-03-29T11:12:01.967 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 10 monclient: _finish_auth 0 2024-03-29T11:12:01.967 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 10 monclient: _check_auth_tickets 2024-03-29T11:12:01.968 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:01.968 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 1 -- 172.21.15.170:0/4223247859 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f55780015c0 con 0x7f55680036e0 2024-03-29T11:12:01.968 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558ca51640 5 monclient: authenticate success, global_id 4106 2024-03-29T11:12:01.968 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558ca51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:01.968 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:01.968 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:01.968 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 10 monclient: dump: 2024-03-29T11:12:01.968 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2024-03-29T11:12:01.968 INFO:teuthology.orchestra.run.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:01.968 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:01.968 INFO:teuthology.orchestra.run.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:01.968 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:01.968 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:01.968 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:01.968 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:01.969 INFO:teuthology.orchestra.run.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:01.969 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-29T11:12:01.969 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558ca51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:01.969 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 1 -- 172.21.15.170:0/4223247859 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5578001790 con 0x7f55680036e0 2024-03-29T11:12:01.969 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:01.969 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 1 -- 172.21.15.170:0/4223247859 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5578001b20 con 0x7f55680036e0 2024-03-29T11:12:01.969 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:01.969 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:01.969 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558559a640 10 monclient: dump: 2024-03-29T11:12:01.969 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2024-03-29T11:12:01.970 INFO:teuthology.orchestra.run.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:01.970 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:01.970 INFO:teuthology.orchestra.run.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:01.970 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:01.970 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:01.970 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:01.970 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:01.971 INFO:teuthology.orchestra.run.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:01.971 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-29T11:12:01.971 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558ca51640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:01.971 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.987+0000 7f558ca51640 4 set_mon_vals no callback set 2024-03-29T11:12:01.972 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.988+0000 7f558ca51640 10 monclient: shutdown 2024-03-29T11:12:01.972 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.988+0000 7f558ca51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:01.972 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.988+0000 7f558ca51640 1 -- 172.21.15.170:0/4223247859 >> v1:172.21.15.170:6790/0 conn(0x7f55680036e0 legacy=0x7f5568005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:01.972 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.988+0000 7f558ca51640 1 -- 172.21.15.170:0/4223247859 shutdown_connections 2024-03-29T11:12:01.972 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.988+0000 7f558ca51640 1 -- 172.21.15.170:0/4223247859 >> 172.21.15.170:0/4223247859 conn(0x7f5588066b00 msgr2=0x7f5588066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:01.972 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.988+0000 7f558ca51640 1 -- 172.21.15.170:0/4223247859 shutdown_connections 2024-03-29T11:12:01.972 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.988+0000 7f558ca51640 1 -- 172.21.15.170:0/4223247859 wait complete. 2024-03-29T11:12:01.978 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.989+0000 7f558ca51640 10 monclient: build_initial_monmap 2024-03-29T11:12:01.978 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.989+0000 7f558ca51640 10 monclient: monmap: 2024-03-29T11:12:01.978 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2024-03-29T11:12:01.978 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:01.978 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2024-03-29T11:12:01.978 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2024-03-29T11:12:01.979 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:01.979 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:01.979 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:01.979 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:12:01.979 INFO:teuthology.orchestra.run.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:12:01.979 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-29T11:12:01.979 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.990+0000 7f558ca51640 1 librados: starting msgr at 2024-03-29T11:12:01.979 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.990+0000 7f558ca51640 1 librados: starting objecter 2024-03-29T11:12:01.979 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.990+0000 7f558ca51640 1 Processor -- start 2024-03-29T11:12:01.979 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.990+0000 7f558ca51640 1 -- start start 2024-03-29T11:12:01.979 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.990+0000 7f558ca51640 1 librados: setting wanted keys 2024-03-29T11:12:01.979 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.990+0000 7f558ca51640 1 librados: calling monclient init 2024-03-29T11:12:01.979 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.990+0000 7f558ca51640 10 monclient: init 2024-03-29T11:12:01.979 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.991+0000 7f558ca51640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:01.979 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.991+0000 7f558ca51640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:12:01.980 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.991+0000 7f558ca51640 10 monclient(hunting): picked mon.noname-c con 0x7f55680036e0 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:01.980 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.991+0000 7f558ca51640 10 monclient(hunting): picked mon.noname-a con 0x7f55680072d0 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:01.980 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.991+0000 7f558ca51640 10 monclient(hunting): picked mon.noname-b con 0x7f558809c8c0 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:01.980 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.991+0000 7f558659c640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f55680036e0 0x7f55881dadc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:53772/0 (socket says 172.21.15.170:53772) 2024-03-29T11:12:01.980 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.991+0000 7f558ca51640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55880af8e0 con 0x7f55680072d0 2024-03-29T11:12:01.980 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.991+0000 7f558659c640 1 -- 172.21.15.170:0/2548543341 learned_addr learned my addr 172.21.15.170:0/2548543341 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:12:01.980 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.991+0000 7f558ca51640 1 -- 172.21.15.170:0/2548543341 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55880afbc0 con 0x7f558809c8c0 2024-03-29T11:12:01.980 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.991+0000 7f558ca51640 1 -- 172.21.15.170:0/2548543341 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55881db4d0 con 0x7f55680036e0 2024-03-29T11:12:01.980 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.991+0000 7f558ca51640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:01.980 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.991+0000 7f5576ffd640 10 client.?.objecter ms_handle_connect 0x7f55680036e0 2024-03-29T11:12:01.980 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.991+0000 7f5576ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:01.980 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.991+0000 7f5576ffd640 10 client.?.objecter ms_handle_connect 0x7f55680072d0 2024-03-29T11:12:01.980 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.991+0000 7f5576ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:01.980 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.991+0000 7f558ca51640 10 monclient(hunting): authenticate will time out at 903.302727s 2024-03-29T11:12:01.981 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.991+0000 7f5576ffd640 10 client.?.objecter ms_handle_connect 0x7f558809c8c0 2024-03-29T11:12:01.981 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.991+0000 7f5576ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:01.981 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.992+0000 7f5586d9d640 1 -- 172.21.15.170:0/2548543341 >> v1:172.21.15.184:6789/0 conn(0x7f558809c8c0 legacy=0x7f55880b0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-29T11:12:01.981 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.992+0000 7f5586d9d640 1 -- 172.21.15.170:0/2548543341 >> v1:172.21.15.184:6789/0 conn(0x7f558809c8c0 legacy=0x7f55880b0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:12:01.981 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.992+0000 7f5586d9d640 1 --1- 172.21.15.170:0/2548543341 >> v1:172.21.15.184:6789/0 conn(0x7f558809c8c0 0x7f55880b0730 :-1 s=OPENED pgs=14 cs=1 l=1).handle_message read tag failed 2024-03-29T11:12:01.981 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.992+0000 7f5586d9d640 1 --1- 172.21.15.170:0/2548543341 >> v1:172.21.15.184:6789/0 conn(0x7f558809c8c0 0x7f55880b0730 :-1 s=OPENED pgs=14 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:12:01.981 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.992+0000 7f5576ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.184:6789/0 2024-03-29T11:12:01.981 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.992+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968962158 0 0) 0x7f55881db4d0 con 0x7f55680036e0 2024-03-29T11:12:01.981 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.993+0000 7f5576ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:01.981 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.993+0000 7f5576ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:01.981 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.993+0000 7f5576ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:01.981 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.993+0000 7f5576ffd640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:12:01.981 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.993+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5568010220 con 0x7f55680036e0 2024-03-29T11:12:01.982 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.993+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1549783034 0 0) 0x7f5568010220 con 0x7f55680036e0 2024-03-29T11:12:01.982 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.993+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55881db4d0 con 0x7f55680036e0 2024-03-29T11:12:01.982 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.993+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5578001ba0 con 0x7f55680036e0 2024-03-29T11:12:01.982 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.993+0000 7f5576ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:01.982 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.993+0000 7f5576ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:01.982 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.993+0000 7f5576ffd640 10 monclient(hunting): dump: 2024-03-29T11:12:01.982 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2024-03-29T11:12:01.982 INFO:teuthology.orchestra.run.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:01.982 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:01.982 INFO:teuthology.orchestra.run.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:01.982 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:01.982 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:01.982 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:01.982 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:01.983 INFO:teuthology.orchestra.run.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:01.983 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-29T11:12:01.983 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.993+0000 7f5576ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:01.983 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.993+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616290155 0 0) 0x7f55880af8e0 con 0x7f55680072d0 2024-03-29T11:12:01.983 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.993+0000 7f5576ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:01.983 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.993+0000 7f5576ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:01.983 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.993+0000 7f5576ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:01.983 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.993+0000 7f5576ffd640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:12:01.983 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.993+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5568010220 con 0x7f55680072d0 2024-03-29T11:12:01.983 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2458100935 0 0) 0x7f5568010220 con 0x7f55680072d0 2024-03-29T11:12:01.983 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55880af8e0 con 0x7f55680072d0 2024-03-29T11:12:01.983 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f557c0018f0 con 0x7f55680072d0 2024-03-29T11:12:01.983 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f5576ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:01.983 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f5576ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:01.983 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f5576ffd640 10 monclient(hunting): dump: 2024-03-29T11:12:01.984 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2024-03-29T11:12:01.984 INFO:teuthology.orchestra.run.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:01.984 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:01.984 INFO:teuthology.orchestra.run.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:01.984 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:01.984 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:01.984 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:01.984 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:01.984 INFO:teuthology.orchestra.run.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:01.984 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-29T11:12:01.984 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3542143727 0 0) 0x7f55880af8e0 con 0x7f55680072d0 2024-03-29T11:12:01.984 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 >> v1:172.21.15.170:6790/0 conn(0x7f55680036e0 legacy=0x7f55881dadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:01.984 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 >> v1:172.21.15.184:6789/0 conn(0x7f558809c8c0 legacy=0x7f55880b0730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:12:01.984 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f5576ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:01.984 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f5576ffd640 1 monclient: found mon.a 2024-03-29T11:12:01.985 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f5576ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:01.985 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f5576ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:01.985 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55881db660 con 0x7f55680072d0 2024-03-29T11:12:01.985 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f5576ffd640 10 monclient: _finish_auth 0 2024-03-29T11:12:01.985 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f5576ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:01.985 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f5576ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:01.985 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f558ca51640 5 monclient: authenticate success, global_id 4113 2024-03-29T11:12:01.985 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f558ca51640 10 monclient: _renew_subs 2024-03-29T11:12:01.985 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f558ca51640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:01.985 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f558ca51640 1 -- 172.21.15.170:0/2548543341 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5588137520 con 0x7f55680072d0 2024-03-29T11:12:01.985 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f558ca51640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:01.985 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f558ca51640 10 monclient: _renew_subs 2024-03-29T11:12:01.985 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f558ca51640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:01.985 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f558ca51640 1 -- 172.21.15.170:0/2548543341 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55880b0e40 con 0x7f55680072d0 2024-03-29T11:12:01.986 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.994+0000 7f558ca51640 1 librados: init done 2024-03-29T11:12:01.986 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.996+0000 7f558ca51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:01.986 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.996+0000 7f558ca51640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:01.986 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.996+0000 7f558ca51640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:01.986 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.996+0000 7f558ca51640 1 -- 172.21.15.170:0/2548543341 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55880b64e0 con 0x7f55680072d0 2024-03-29T11:12:01.986 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.996+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f557c001d60 con 0x7f55680072d0 2024-03-29T11:12:01.986 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.996+0000 7f5576ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:01.986 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.996+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f557c004980 con 0x7f55680072d0 2024-03-29T11:12:01.986 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.996+0000 7f5576ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:01.986 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.996+0000 7f5576ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:01.986 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.996+0000 7f5576ffd640 10 monclient: dump: 2024-03-29T11:12:01.986 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2024-03-29T11:12:01.986 INFO:teuthology.orchestra.run.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:01.986 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:01.986 INFO:teuthology.orchestra.run.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:01.987 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:01.987 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:01.987 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:01.987 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:01.987 INFO:teuthology.orchestra.run.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:01.987 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-29T11:12:01.987 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.996+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f557c004ef0 con 0x7f55680072d0 2024-03-29T11:12:01.987 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.996+0000 7f5584d99640 4 set_mon_vals no callback set 2024-03-29T11:12:01.987 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.996+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 27679067 0 0) 0x7f557c0064c0 con 0x7f55680072d0 2024-03-29T11:12:01.987 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.996+0000 7f5576ffd640 10 client.4113.objecter ms_dispatch 0x7f5588000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:12:01.987 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.996+0000 7f5576ffd640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:12:01.987 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.996+0000 7f5576ffd640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:12:01.987 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.996+0000 7f5576ffd640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-29T11:12:01.987 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.998+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f557c006840 con 0x7f55680072d0 2024-03-29T11:12:01.987 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.999+0000 7f5576ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:01.988 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:01.999+0000 7f5576ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:02.048 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:02.071+0000 7f558ca51640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:12:02.048 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:02.071+0000 7f558ca51640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:12:02.048 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:02.071+0000 7f558ca51640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:02.049 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:02.071+0000 7f558ca51640 1 -- 172.21.15.170:0/2548543341 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f558804f730 con 0x7f55680072d0 2024-03-29T11:12:02.049 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:02.072+0000 7f5576ffd640 1 -- 172.21.15.170:0/2548543341 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f557c005280 con 0x7f55680072d0 2024-03-29T11:12:02.049 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:02.072+0000 7f5576ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:12:02.050 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:02.072+0000 7f5576ffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:12:02.050 INFO:teuthology.orchestra.run.smithi170.stderr:adjusted tunables profile to default 2024-03-29T11:12:02.052 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:02.073+0000 7f558ca51640 10 librados: watch_flush enter 2024-03-29T11:12:02.053 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:02.073+0000 7f558ca51640 10 librados: watch_flush exit 2024-03-29T11:12:02.053 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:02.073+0000 7f558ca51640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-29T11:12:02.053 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:02.073+0000 7f558ca51640 10 client.4113.objecter successfully canceled tick 2024-03-29T11:12:02.053 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:02.073+0000 7f558ca51640 10 monclient: shutdown 2024-03-29T11:12:02.053 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:02.073+0000 7f558ca51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:02.053 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:02.073+0000 7f558ca51640 1 -- 172.21.15.170:0/2548543341 >> v1:172.21.15.170:6789/0 conn(0x7f55680072d0 legacy=0x7f5588136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:02.053 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:02.073+0000 7f558ca51640 1 -- 172.21.15.170:0/2548543341 shutdown_connections 2024-03-29T11:12:02.053 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:02.073+0000 7f558ca51640 1 -- 172.21.15.170:0/2548543341 >> 172.21.15.170:0/2548543341 conn(0x7f5588066b00 msgr2=0x7f558806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:02.053 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:02.073+0000 7f558ca51640 1 -- 172.21.15.170:0/2548543341 shutdown_connections 2024-03-29T11:12:02.053 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:02.073+0000 7f558ca51640 1 -- 172.21.15.170:0/2548543341 wait complete. 2024-03-29T11:12:02.053 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:02.073+0000 7f558ca51640 1 librados: shutdown 2024-03-29T11:12:02.062 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:12:02.062 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:12:02.063 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:12:02.063 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:12:02.091 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:12:02.091 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:12:02.158 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:12:02.158 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:12:02.225 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-29T11:12:02.225 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:12:02.292 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-29T11:12:02.292 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:12:02.319 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-29T11:12:02.319 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:12:02.387 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-29T11:12:02.387 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:12:02.455 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-29T11:12:02.455 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:12:02.480 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:02.502+0000 7f84d7221040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:12:02.483 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:02.504+0000 7f1e7ea43040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:12:02.509 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 42e2bbec-c0e5-45c1-a2c1-55829916ae2c 0 2024-03-29T11:12:02.532 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:02.555+0000 7f84d7221040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:12:02.539 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:02.561+0000 7f1e7ea43040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:12:02.639 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.661+0000 7f138c00d640 10 monclient: get_monmap_and_config 2024-03-29T11:12:02.639 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.661+0000 7f138c00d640 10 monclient: build_initial_monmap 2024-03-29T11:12:02.639 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.661+0000 7f138c00d640 10 monclient: monmap: 2024-03-29T11:12:02.639 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2024-03-29T11:12:02.640 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:02.640 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:12:02.662411+0000 2024-03-29T11:12:02.640 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:12:02.662411+0000 2024-03-29T11:12:02.640 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:02.640 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:02.640 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:02.640 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:12:02.640 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:12:02.640 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:02.640 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.661+0000 7f138c00d640 1 Processor -- start 2024-03-29T11:12:02.652 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.661+0000 7f138c00d640 1 -- start start 2024-03-29T11:12:02.653 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.661+0000 7f138c00d640 10 monclient: init 2024-03-29T11:12:02.653 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f138c00d640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:02.653 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f138c00d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:02.653 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f138c00d640 10 monclient(hunting): picked mon.noname-c con 0x7f138409ac40 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:02.653 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f138c00d640 10 monclient(hunting): picked mon.noname-a con 0x7f138409c870 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:02.653 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f138c00d640 10 monclient(hunting): picked mon.noname-b con 0x7f13840b4360 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:02.653 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f138c00d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f138409dec0 con 0x7f138409c870 2024-03-29T11:12:02.653 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f138c00d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f138409da30 con 0x7f138409ac40 2024-03-29T11:12:02.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f138c00d640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13840b39c0 con 0x7f13840b4360 2024-03-29T11:12:02.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f138c00d640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:02.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f138c00d640 10 monclient(hunting): authenticate will time out at 912.846492s 2024-03-29T11:12:02.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f1389da9640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f138409ac40 0x7f13840b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:59456/0 (socket says 172.21.15.184:59456) 2024-03-29T11:12:02.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f1389da9640 1 -- 172.21.15.184:0/3748284621 learned_addr learned my addr 172.21.15.184:0/3748284621 (peer_addr_for_me v1:172.21.15.184:0/0) 2024-03-29T11:12:02.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f1388da7640 1 -- 172.21.15.184:0/3748284621 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414947664 0 0) 0x7f138409da30 con 0x7f138409ac40 2024-03-29T11:12:02.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f1388da7640 10 monclient(hunting): _negotiate 2024-03-29T11:12:02.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f1388da7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:02.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f1388da7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:02.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f1388da7640 10 monclient(hunting): my global_id is 4102 2024-03-29T11:12:02.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f1388da7640 1 -- 172.21.15.184:0/3748284621 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1370003520 con 0x7f138409ac40 2024-03-29T11:12:02.655 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f1388da7640 1 -- 172.21.15.184:0/3748284621 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615368427 0 0) 0x7f1380001110 con 0x7f138409c870 2024-03-29T11:12:02.655 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f1388da7640 10 monclient(hunting): _negotiate 2024-03-29T11:12:02.655 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f1388da7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:02.655 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f1388da7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:02.655 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f1388da7640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:12:02.655 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.662+0000 7f1388da7640 1 -- 172.21.15.184:0/3748284621 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f138409da30 con 0x7f138409c870 2024-03-29T11:12:02.655 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 1 -- 172.21.15.184:0/3748284621 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951952152 0 0) 0x7f13840b39c0 con 0x7f13840b4360 2024-03-29T11:12:02.655 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 10 monclient(hunting): _negotiate 2024-03-29T11:12:02.656 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:02.656 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:02.656 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:12:02.656 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 1 -- 172.21.15.184:0/3748284621 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1380001110 con 0x7f13840b4360 2024-03-29T11:12:02.656 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 1 -- 172.21.15.184:0/3748284621 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 44769273 0 0) 0x7f1370003520 con 0x7f138409ac40 2024-03-29T11:12:02.657 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 1 -- 172.21.15.184:0/3748284621 >> v1:172.21.15.170:6790/0 conn(0x7f13840b4360 legacy=0x7f138414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:02.657 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 1 -- 172.21.15.184:0/3748284621 >> v1:172.21.15.170:6789/0 conn(0x7f138409c870 legacy=0x7f138413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:02.657 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 10 monclient: _finish_hunting 0 2024-03-29T11:12:02.657 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 1 monclient: found mon.noname-c 2024-03-29T11:12:02.657 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:02.657 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:12:02.657 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 1 -- 172.21.15.184:0/3748284621 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13840b5160 con 0x7f138409ac40 2024-03-29T11:12:02.657 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 10 monclient: _finish_auth 0 2024-03-29T11:12:02.657 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 10 monclient: _check_auth_tickets 2024-03-29T11:12:02.657 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:02.657 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 1 -- 172.21.15.184:0/3748284621 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f1374001690 con 0x7f138409ac40 2024-03-29T11:12:02.657 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:02.657 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:02.657 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 10 monclient: dump: 2024-03-29T11:12:02.658 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:02.658 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:02.658 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.658 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.658 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:02.658 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:02.658 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:02.658 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:02.658 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:02.658 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:02.658 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f138c00d640 5 monclient: authenticate success, global_id 4102 2024-03-29T11:12:02.658 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f138c00d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:02.658 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 1 -- 172.21.15.184:0/3748284621 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1374001870 con 0x7f138409ac40 2024-03-29T11:12:02.658 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:02.659 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 1 -- 172.21.15.184:0/3748284621 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f1374001c80 con 0x7f138409ac40 2024-03-29T11:12:02.659 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f138c00d640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:02.659 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f138c00d640 4 set_mon_vals no callback set 2024-03-29T11:12:02.659 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:02.659 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:02.659 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f1388da7640 10 monclient: dump: 2024-03-29T11:12:02.659 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:02.659 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:02.659 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.659 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.659 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:02.659 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:02.659 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:02.659 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:02.659 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:02.659 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:02.660 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f138c00d640 10 monclient: shutdown 2024-03-29T11:12:02.660 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f138c00d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:02.660 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f138c00d640 1 -- 172.21.15.184:0/3748284621 >> v1:172.21.15.184:6789/0 conn(0x7f138409ac40 legacy=0x7f13840b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:02.660 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f138c00d640 1 -- 172.21.15.184:0/3748284621 shutdown_connections 2024-03-29T11:12:02.660 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f138c00d640 1 -- 172.21.15.184:0/3748284621 >> 172.21.15.184:0/3748284621 conn(0x7f1384066b00 msgr2=0x7f1384066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:02.660 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f138c00d640 1 -- 172.21.15.184:0/3748284621 shutdown_connections 2024-03-29T11:12:02.660 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.663+0000 7f138c00d640 1 -- 172.21.15.184:0/3748284621 wait complete. 2024-03-29T11:12:02.660 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.664+0000 7f138c00d640 10 monclient: build_initial_monmap 2024-03-29T11:12:02.660 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.664+0000 7f138c00d640 10 monclient: monmap: 2024-03-29T11:12:02.660 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2024-03-29T11:12:02.660 INFO:teuthology.orchestra.run.smithi184.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:02.660 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 0.000000 2024-03-29T11:12:02.660 INFO:teuthology.orchestra.run.smithi184.stderr:created 0.000000 2024-03-29T11:12:02.660 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:02.661 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:02.661 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:02.661 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:12:02.661 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:12:02.661 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:02.661 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.664+0000 7f138c00d640 1 librados: starting msgr at 2024-03-29T11:12:02.661 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.664+0000 7f138c00d640 1 librados: starting objecter 2024-03-29T11:12:02.661 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.664+0000 7f138c00d640 1 Processor -- start 2024-03-29T11:12:02.661 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.664+0000 7f138c00d640 1 -- start start 2024-03-29T11:12:02.661 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.664+0000 7f138c00d640 1 librados: setting wanted keys 2024-03-29T11:12:02.661 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.664+0000 7f138c00d640 1 librados: calling monclient init 2024-03-29T11:12:02.661 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.664+0000 7f138c00d640 10 monclient: init 2024-03-29T11:12:02.661 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f138c00d640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:02.661 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f138c00d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:12:02.661 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f138c00d640 10 monclient(hunting): picked mon.noname-b con 0x7f138409ac40 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:02.661 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f138c00d640 10 monclient(hunting): picked mon.noname-c con 0x7f13840acd00 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:02.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f138c00d640 10 monclient(hunting): picked mon.noname-a con 0x7f13841425b0 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:02.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f138c00d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1384142c10 con 0x7f13841425b0 2024-03-29T11:12:02.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f138c00d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13840adf60 con 0x7f138409ac40 2024-03-29T11:12:02.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f138c00d640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f138413b770 con 0x7f13840acd00 2024-03-29T11:12:02.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f138c00d640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:02.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f138c00d640 10 monclient(hunting): authenticate will time out at 912.849508s 2024-03-29T11:12:02.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f1389da9640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f138409ac40 0x7f138413b060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:59460/0 (socket says 172.21.15.184:59460) 2024-03-29T11:12:02.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f1389da9640 1 -- 172.21.15.184:0/3378380832 learned_addr learned my addr 172.21.15.184:0/3378380832 (peer_addr_for_me v1:172.21.15.184:0/0) 2024-03-29T11:12:02.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 10 client.?.objecter ms_handle_connect 0x7f138409ac40 2024-03-29T11:12:02.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:02.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 10 client.?.objecter ms_handle_connect 0x7f13840acd00 2024-03-29T11:12:02.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:02.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 10 client.?.objecter ms_handle_connect 0x7f13841425b0 2024-03-29T11:12:02.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:02.663 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707259293 0 0) 0x7f13840adf60 con 0x7f138409ac40 2024-03-29T11:12:02.663 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:02.663 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:02.663 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:02.663 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:12:02.663 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1368004eb0 con 0x7f138409ac40 2024-03-29T11:12:02.663 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 867213063 0 0) 0x7f138413b770 con 0x7f13840acd00 2024-03-29T11:12:02.663 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:02.664 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:02.664 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:02.664 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:12:02.664 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13840adf60 con 0x7f13840acd00 2024-03-29T11:12:02.664 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277119536 0 0) 0x7f1384142c10 con 0x7f13841425b0 2024-03-29T11:12:02.664 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:02.664 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:02.664 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:02.664 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:12:02.664 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f138413b770 con 0x7f13841425b0 2024-03-29T11:12:02.664 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2221593251 0 0) 0x7f1368004eb0 con 0x7f138409ac40 2024-03-29T11:12:02.665 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.665+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1384142c10 con 0x7f138409ac40 2024-03-29T11:12:02.665 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f1374001bc0 con 0x7f138409ac40 2024-03-29T11:12:02.665 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:02.665 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:02.665 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 10 monclient(hunting): dump: 2024-03-29T11:12:02.665 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:02.665 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:02.665 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.666 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.666 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:02.666 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:02.666 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:02.666 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:02.666 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:02.666 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:02.666 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:02.666 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1603187458 0 0) 0x7f138413b770 con 0x7f13841425b0 2024-03-29T11:12:02.666 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1368004eb0 con 0x7f13841425b0 2024-03-29T11:12:02.667 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f137c001bd0 con 0x7f13841425b0 2024-03-29T11:12:02.667 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:02.667 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:02.667 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 10 monclient(hunting): dump: 2024-03-29T11:12:02.667 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:02.667 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:02.667 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.667 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.667 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:02.667 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:02.667 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:02.667 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:02.667 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:02.668 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:02.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2185546401 0 0) 0x7f13840adf60 con 0x7f13840acd00 2024-03-29T11:12:02.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f138413b770 con 0x7f13840acd00 2024-03-29T11:12:02.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f13800018a0 con 0x7f13840acd00 2024-03-29T11:12:02.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:02.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:02.670 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 10 monclient(hunting): dump: 2024-03-29T11:12:02.671 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:02.671 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:02.671 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.671 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.671 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:02.671 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:02.671 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:02.671 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:02.671 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:02.671 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:02.671 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2576605046 0 0) 0x7f1384142c10 con 0x7f138409ac40 2024-03-29T11:12:02.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 >> v1:172.21.15.170:6790/0 conn(0x7f13840acd00 legacy=0x7f138413ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:02.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 >> v1:172.21.15.170:6789/0 conn(0x7f13841425b0 legacy=0x7f13841e2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:02.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:12:02.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 1 monclient: found mon.b 2024-03-29T11:12:02.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:02.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:02.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138413b900 con 0x7f138409ac40 2024-03-29T11:12:02.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:12:02.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:12:02.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:02.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f138c00d640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:12:02.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f138c00d640 10 monclient: _renew_subs 2024-03-29T11:12:02.673 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f138c00d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:02.673 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f138c00d640 1 -- 172.21.15.184:0/3378380832 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13840b7020 con 0x7f138409ac40 2024-03-29T11:12:02.673 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f138c00d640 10 client.4105.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:02.673 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f138c00d640 10 monclient: _renew_subs 2024-03-29T11:12:02.673 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f138c00d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:02.673 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f138c00d640 1 -- 172.21.15.184:0/3378380832 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13841406b0 con 0x7f138409ac40 2024-03-29T11:12:02.673 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f138c00d640 1 librados: init done 2024-03-29T11:12:02.674 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1374004bf0 con 0x7f138409ac40 2024-03-29T11:12:02.674 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:02.674 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f1374004f00 con 0x7f138409ac40 2024-03-29T11:12:02.674 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:02.674 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:02.674 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137bfff640 4 set_mon_vals no callback set 2024-03-29T11:12:02.674 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 10 monclient: dump: 2024-03-29T11:12:02.674 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:02.674 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:02.674 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.674 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.674 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:02.675 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:02.675 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:02.675 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:02.675 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:02.675 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:02.675 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f13740053f0 con 0x7f138409ac40 2024-03-29T11:12:02.675 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1610751693 0 0) 0x7f13740067d0 con 0x7f138409ac40 2024-03-29T11:12:02.675 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 10 client.4105.objecter ms_dispatch 0x7f1384000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:12:02.675 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 3 client.4105.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:12:02.675 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 3 client.4105.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:12:02.675 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.666+0000 7f137a7fc640 20 client.4105.objecter dump_active .. 0 homeless 2024-03-29T11:12:02.675 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.667+0000 7f138c00d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:02.675 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.667+0000 7f138c00d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:02.676 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.667+0000 7f138c00d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:02.676 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.667+0000 7f138c00d640 1 -- 172.21.15.184:0/3378380832 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f136c005080 con 0x7f138409ac40 2024-03-29T11:12:02.676 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.671+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1374006a10 con 0x7f138409ac40 2024-03-29T11:12:02.676 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.671+0000 7f137a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:02.676 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.671+0000 7f137a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:02.721 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.743+0000 7f138c00d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "42e2bbec-c0e5-45c1-a2c1-55829916ae2c", "id": 0}] 2024-03-29T11:12:02.721 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.743+0000 7f138c00d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "42e2bbec-c0e5-45c1-a2c1-55829916ae2c", "id": 0}] 2024-03-29T11:12:02.722 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.743+0000 7f138c00d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:02.722 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.743+0000 7f138c00d640 1 -- 172.21.15.184:0/3378380832 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd new", "uuid": "42e2bbec-c0e5-45c1-a2c1-55829916ae2c", "id": 0} v 0) v1 -- 0x7f136c0058b0 con 0x7f138409ac40 2024-03-29T11:12:02.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.749+0000 7f137a7fc640 1 -- 172.21.15.184:0/3378380832 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "42e2bbec-c0e5-45c1-a2c1-55829916ae2c", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3751232329 0 732707515) 0x7f1374005860 con 0x7f138409ac40 2024-03-29T11:12:02.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.749+0000 7f137a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "42e2bbec-c0e5-45c1-a2c1-55829916ae2c", "id": 0}] 2024-03-29T11:12:02.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.749+0000 7f137a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:02.728 INFO:teuthology.orchestra.run.smithi184.stdout:0 2024-03-29T11:12:02.730 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.750+0000 7f138c00d640 10 librados: watch_flush enter 2024-03-29T11:12:02.730 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.750+0000 7f138c00d640 10 librados: watch_flush exit 2024-03-29T11:12:02.730 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.750+0000 7f138c00d640 20 client.4105.objecter shutdown clearing up homeless session... 2024-03-29T11:12:02.730 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.750+0000 7f138c00d640 10 client.4105.objecter successfully canceled tick 2024-03-29T11:12:02.730 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.750+0000 7f138c00d640 10 monclient: shutdown 2024-03-29T11:12:02.730 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.750+0000 7f138c00d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:02.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.750+0000 7f138c00d640 1 -- 172.21.15.184:0/3378380832 >> v1:172.21.15.184:6789/0 conn(0x7f138409ac40 legacy=0x7f138413b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:02.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.750+0000 7f138c00d640 1 -- 172.21.15.184:0/3378380832 shutdown_connections 2024-03-29T11:12:02.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.750+0000 7f138c00d640 1 -- 172.21.15.184:0/3378380832 >> 172.21.15.184:0/3378380832 conn(0x7f1384066b00 msgr2=0x7f138413d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:02.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.750+0000 7f138c00d640 1 -- 172.21.15.184:0/3378380832 shutdown_connections 2024-03-29T11:12:02.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.750+0000 7f138c00d640 1 -- 172.21.15.184:0/3378380832 wait complete. 2024-03-29T11:12:02.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.750+0000 7f138c00d640 1 librados: shutdown 2024-03-29T11:12:02.739 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new dcbc3485-c2c7-44a0-8cc5-648b65ebe62f 1 2024-03-29T11:12:02.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.861+0000 7fb52f0e6640 10 monclient: get_monmap_and_config 2024-03-29T11:12:02.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.861+0000 7fb52f0e6640 10 monclient: build_initial_monmap 2024-03-29T11:12:02.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.861+0000 7fb52f0e6640 10 monclient: monmap: 2024-03-29T11:12:02.839 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2024-03-29T11:12:02.839 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:02.839 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:12:02.862339+0000 2024-03-29T11:12:02.839 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:12:02.862339+0000 2024-03-29T11:12:02.839 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:02.839 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:02.840 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:02.840 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:12:02.840 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:12:02.840 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:02.849 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.861+0000 7fb52f0e6640 1 Processor -- start 2024-03-29T11:12:02.849 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.861+0000 7fb52f0e6640 1 -- start start 2024-03-29T11:12:02.849 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.861+0000 7fb52f0e6640 10 monclient: init 2024-03-29T11:12:02.849 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb52f0e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:02.849 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb52f0e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:12:02.849 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb52f0e6640 10 monclient(hunting): picked mon.noname-b con 0x7fb5280b0ad0 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:02.849 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb52f0e6640 10 monclient(hunting): picked mon.noname-a con 0x7fb528142320 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:02.849 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb52f0e6640 10 monclient(hunting): picked mon.noname-c con 0x7fb52806f430 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:02.849 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb52f0e6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb528142780 con 0x7fb528142320 2024-03-29T11:12:02.849 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb52f0e6640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb528072690 con 0x7fb52806f430 2024-03-29T11:12:02.849 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb52f0e6640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb528071230 con 0x7fb5280b0ad0 2024-03-29T11:12:02.850 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb52f0e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:02.850 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb52f0e6640 10 monclient(hunting): authenticate will time out at 913.046477s 2024-03-29T11:12:02.850 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb52d683640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fb52806f430 0x7fb52814a6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:59470/0 (socket says 172.21.15.184:59470) 2024-03-29T11:12:02.850 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb52d683640 1 -- 172.21.15.184:0/3314644686 learned_addr learned my addr 172.21.15.184:0/3314644686 (peer_addr_for_me v1:172.21.15.184:0/0) 2024-03-29T11:12:02.850 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb5277fe640 1 -- 172.21.15.184:0/3314644686 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050650681 0 0) 0x7fb528072690 con 0x7fb52806f430 2024-03-29T11:12:02.850 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb5277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:02.850 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb5277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:02.850 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb5277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:02.850 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb5277fe640 10 monclient(hunting): my global_id is 4108 2024-03-29T11:12:02.850 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb5277fe640 1 -- 172.21.15.184:0/3314644686 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb518003520 con 0x7fb52806f430 2024-03-29T11:12:02.850 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb5277fe640 1 -- 172.21.15.184:0/3314644686 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067130742 0 0) 0x7fb528142780 con 0x7fb528142320 2024-03-29T11:12:02.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb5277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:02.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb5277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:02.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb5277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:02.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb5277fe640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:12:02.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb5277fe640 1 -- 172.21.15.184:0/3314644686 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb528072690 con 0x7fb528142320 2024-03-29T11:12:02.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb5277fe640 1 -- 172.21.15.184:0/3314644686 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430827894 0 0) 0x7fb528071230 con 0x7fb5280b0ad0 2024-03-29T11:12:02.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb5277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:02.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb5277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:02.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb5277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:02.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb5277fe640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:12:02.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.862+0000 7fb5277fe640 1 -- 172.21.15.184:0/3314644686 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb528142780 con 0x7fb5280b0ad0 2024-03-29T11:12:02.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 1 -- 172.21.15.184:0/3314644686 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 889220888 0 0) 0x7fb528072690 con 0x7fb528142320 2024-03-29T11:12:02.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 1 -- 172.21.15.184:0/3314644686 >> v1:172.21.15.170:6790/0 conn(0x7fb5280b0ad0 legacy=0x7fb528071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:02.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 1 -- 172.21.15.184:0/3314644686 >> v1:172.21.15.184:6789/0 conn(0x7fb52806f430 legacy=0x7fb52814a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:02.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 10 monclient: _finish_hunting 0 2024-03-29T11:12:02.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 1 monclient: found mon.noname-a 2024-03-29T11:12:02.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:02.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:12:02.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 1 -- 172.21.15.184:0/3314644686 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5280714a0 con 0x7fb528142320 2024-03-29T11:12:02.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 10 monclient: _finish_auth 0 2024-03-29T11:12:02.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 10 monclient: _check_auth_tickets 2024-03-29T11:12:02.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:02.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 1 -- 172.21.15.184:0/3314644686 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb514001670 con 0x7fb528142320 2024-03-29T11:12:02.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:02.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:02.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 10 monclient: dump: 2024-03-29T11:12:02.852 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:02.852 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:02.852 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.853 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.853 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:02.853 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:02.853 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:02.853 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:02.853 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:02.853 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:02.853 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb52f0e6640 5 monclient: authenticate success, global_id 4122 2024-03-29T11:12:02.853 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb52f0e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:02.853 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 1 -- 172.21.15.184:0/3314644686 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5140017d0 con 0x7fb528142320 2024-03-29T11:12:02.853 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:02.853 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 1 -- 172.21.15.184:0/3314644686 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb514001c20 con 0x7fb528142320 2024-03-29T11:12:02.853 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb52f0e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:02.853 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb52f0e6640 4 set_mon_vals no callback set 2024-03-29T11:12:02.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:02.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:02.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb5277fe640 10 monclient: dump: 2024-03-29T11:12:02.854 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:02.854 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:02.854 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.854 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.854 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:02.854 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:02.854 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:02.854 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:02.854 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:02.854 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:02.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb52f0e6640 10 monclient: shutdown 2024-03-29T11:12:02.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb52f0e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:02.855 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb52f0e6640 1 -- 172.21.15.184:0/3314644686 >> v1:172.21.15.170:6789/0 conn(0x7fb528142320 legacy=0x7fb52806ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:02.855 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb52f0e6640 1 -- 172.21.15.184:0/3314644686 shutdown_connections 2024-03-29T11:12:02.855 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb52f0e6640 1 -- 172.21.15.184:0/3314644686 >> 172.21.15.184:0/3314644686 conn(0x7fb5280b4970 msgr2=0x7fb5280b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:02.855 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb52f0e6640 1 -- 172.21.15.184:0/3314644686 shutdown_connections 2024-03-29T11:12:02.855 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.863+0000 7fb52f0e6640 1 -- 172.21.15.184:0/3314644686 wait complete. 2024-03-29T11:12:02.855 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.864+0000 7fb52f0e6640 10 monclient: build_initial_monmap 2024-03-29T11:12:02.855 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.864+0000 7fb52f0e6640 10 monclient: monmap: 2024-03-29T11:12:02.855 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2024-03-29T11:12:02.855 INFO:teuthology.orchestra.run.smithi184.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:02.855 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 0.000000 2024-03-29T11:12:02.855 INFO:teuthology.orchestra.run.smithi184.stderr:created 0.000000 2024-03-29T11:12:02.855 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:02.855 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:02.855 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:02.856 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:12:02.856 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:12:02.856 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:02.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.864+0000 7fb52f0e6640 1 librados: starting msgr at 2024-03-29T11:12:02.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.864+0000 7fb52f0e6640 1 librados: starting objecter 2024-03-29T11:12:02.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.864+0000 7fb52f0e6640 1 Processor -- start 2024-03-29T11:12:02.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.864+0000 7fb52f0e6640 1 -- start start 2024-03-29T11:12:02.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.864+0000 7fb52f0e6640 1 librados: setting wanted keys 2024-03-29T11:12:02.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.864+0000 7fb52f0e6640 1 librados: calling monclient init 2024-03-29T11:12:02.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.864+0000 7fb52f0e6640 10 monclient: init 2024-03-29T11:12:02.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb52f0e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:02.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb52f0e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:12:02.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb52f0e6640 10 monclient(hunting): picked mon.noname-c con 0x7fb52806f430 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:02.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb52f0e6640 10 monclient(hunting): picked mon.noname-b con 0x7fb5280b0ad0 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:02.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb52f0e6640 10 monclient(hunting): picked mon.noname-a con 0x7fb528142320 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:02.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb52f0e6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5280aa4c0 con 0x7fb528142320 2024-03-29T11:12:02.857 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb52f0e6640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5280ac1a0 con 0x7fb5280b0ad0 2024-03-29T11:12:02.857 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb52f0e6640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5280ab010 con 0x7fb52806f430 2024-03-29T11:12:02.857 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb52f0e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:02.857 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb52f0e6640 10 monclient(hunting): authenticate will time out at 913.049601s 2024-03-29T11:12:02.857 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb527fff640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fb5280b0ad0 0x7fb5280adab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:59476/0 (socket says 172.21.15.184:59476) 2024-03-29T11:12:02.857 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb527fff640 1 -- 172.21.15.184:0/3993934927 learned_addr learned my addr 172.21.15.184:0/3993934927 (peer_addr_for_me v1:172.21.15.184:0/0) 2024-03-29T11:12:02.857 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb5257fa640 10 client.?.objecter ms_handle_connect 0x7fb5280b0ad0 2024-03-29T11:12:02.857 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb5257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:02.857 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb5257fa640 10 client.?.objecter ms_handle_connect 0x7fb52806f430 2024-03-29T11:12:02.857 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb5257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:02.857 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb5257fa640 10 client.?.objecter ms_handle_connect 0x7fb528142320 2024-03-29T11:12:02.857 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb5257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:02.857 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967621943 0 0) 0x7fb5280ac1a0 con 0x7fb5280b0ad0 2024-03-29T11:12:02.857 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb5257fa640 10 monclient(hunting): _negotiate 2024-03-29T11:12:02.857 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb5257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:02.858 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb5257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:02.858 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb5257fa640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:12:02.858 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb50c004eb0 con 0x7fb5280b0ad0 2024-03-29T11:12:02.858 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.865+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98534960 0 0) 0x7fb5280ab010 con 0x7fb52806f430 2024-03-29T11:12:02.858 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient(hunting): _negotiate 2024-03-29T11:12:02.858 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:02.858 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:02.858 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:12:02.858 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5280ac1a0 con 0x7fb52806f430 2024-03-29T11:12:02.858 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338287174 0 0) 0x7fb5280aa4c0 con 0x7fb528142320 2024-03-29T11:12:02.858 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient(hunting): _negotiate 2024-03-29T11:12:02.858 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:02.858 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:02.858 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:12:02.859 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5280ab010 con 0x7fb528142320 2024-03-29T11:12:02.859 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 998648029 0 0) 0x7fb50c004eb0 con 0x7fb5280b0ad0 2024-03-29T11:12:02.859 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5280aa4c0 con 0x7fb5280b0ad0 2024-03-29T11:12:02.859 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb5140035f0 con 0x7fb5280b0ad0 2024-03-29T11:12:02.859 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:02.859 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:02.859 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient(hunting): dump: 2024-03-29T11:12:02.859 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:02.859 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:02.859 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.859 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.859 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:02.859 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:02.859 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:02.859 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:02.860 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:02.860 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:02.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:02.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3805655653 0 0) 0x7fb5280ab010 con 0x7fb528142320 2024-03-29T11:12:02.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb50c004eb0 con 0x7fb528142320 2024-03-29T11:12:02.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb520001410 con 0x7fb528142320 2024-03-29T11:12:02.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:02.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:02.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient(hunting): dump: 2024-03-29T11:12:02.860 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:02.860 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:02.860 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.860 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.860 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:02.860 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:02.861 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:02.861 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:02.861 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:02.861 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:02.861 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 664985970 0 0) 0x7fb5280ac1a0 con 0x7fb52806f430 2024-03-29T11:12:02.861 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5280ab010 con 0x7fb52806f430 2024-03-29T11:12:02.861 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb51c001870 con 0x7fb52806f430 2024-03-29T11:12:02.861 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:02.861 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:02.861 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient(hunting): dump: 2024-03-29T11:12:02.861 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:02.861 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:02.861 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.861 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.862 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:02.862 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:02.862 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:02.862 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:02.862 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:02.862 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:02.862 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3056919180 0 0) 0x7fb5280aa4c0 con 0x7fb5280b0ad0 2024-03-29T11:12:02.862 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 >> v1:172.21.15.170:6790/0 conn(0x7fb52806f430 legacy=0x7fb5280a9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:02.862 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 >> v1:172.21.15.170:6789/0 conn(0x7fb528142320 legacy=0x7fb5281d30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:02.862 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient: _finish_hunting 0 2024-03-29T11:12:02.862 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 monclient: found mon.b 2024-03-29T11:12:02.862 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:02.862 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:02.862 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52813fc40 con 0x7fb5280b0ad0 2024-03-29T11:12:02.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient: _finish_auth 0 2024-03-29T11:12:02.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient: _check_auth_tickets 2024-03-29T11:12:02.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:02.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb52f0e6640 5 monclient: authenticate success, global_id 4111 2024-03-29T11:12:02.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb52f0e6640 10 monclient: _renew_subs 2024-03-29T11:12:02.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb52f0e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:02.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb52f0e6640 1 -- 172.21.15.184:0/3993934927 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5280aab30 con 0x7fb5280b0ad0 2024-03-29T11:12:02.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb52f0e6640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:02.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb52f0e6640 10 monclient: _renew_subs 2024-03-29T11:12:02.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb52f0e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:02.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb52f0e6640 1 -- 172.21.15.184:0/3993934927 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5280ae3a0 con 0x7fb5280b0ad0 2024-03-29T11:12:02.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb52f0e6640 1 librados: init done 2024-03-29T11:12:02.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb514003a00 con 0x7fb5280b0ad0 2024-03-29T11:12:02.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:02.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb514003d90 con 0x7fb5280b0ad0 2024-03-29T11:12:02.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:02.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:02.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb526ffd640 4 set_mon_vals no callback set 2024-03-29T11:12:02.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 monclient: dump: 2024-03-29T11:12:02.864 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:02.864 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:02.864 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.864 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:02.864 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:02.865 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:02.865 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:02.865 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:02.865 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:02.865 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:02.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb514004300 con 0x7fb5280b0ad0 2024-03-29T11:12:02.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2704707302 0 0) 0x7fb514006740 con 0x7fb5280b0ad0 2024-03-29T11:12:02.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 10 client.4111.objecter ms_dispatch 0x7fb528000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:12:02.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 3 client.4111.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:12:02.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 3 client.4111.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:12:02.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.866+0000 7fb5257fa640 20 client.4111.objecter dump_active .. 0 homeless 2024-03-29T11:12:02.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.867+0000 7fb52f0e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:02.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.867+0000 7fb52f0e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:02.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.867+0000 7fb52f0e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:02.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.867+0000 7fb52f0e6640 1 -- 172.21.15.184:0/3993934927 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb508005080 con 0x7fb5280b0ad0 2024-03-29T11:12:02.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.869+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb514006a20 con 0x7fb5280b0ad0 2024-03-29T11:12:02.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.869+0000 7fb5257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:02.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.869+0000 7fb5257fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:02.919 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.940+0000 7fb52f0e6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dcbc3485-c2c7-44a0-8cc5-648b65ebe62f", "id": 1}] 2024-03-29T11:12:02.919 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.940+0000 7fb52f0e6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dcbc3485-c2c7-44a0-8cc5-648b65ebe62f", "id": 1}] 2024-03-29T11:12:02.919 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.940+0000 7fb52f0e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:02.919 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.940+0000 7fb52f0e6640 1 -- 172.21.15.184:0/3993934927 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd new", "uuid": "dcbc3485-c2c7-44a0-8cc5-648b65ebe62f", "id": 1} v 0) v1 -- 0x7fb5080058b0 con 0x7fb5280b0ad0 2024-03-29T11:12:02.926 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.948+0000 7fb5257fa640 1 -- 172.21.15.184:0/3993934927 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dcbc3485-c2c7-44a0-8cc5-648b65ebe62f", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3632752378 0 940483276) 0x7fb514006bc0 con 0x7fb5280b0ad0 2024-03-29T11:12:02.927 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.948+0000 7fb5257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dcbc3485-c2c7-44a0-8cc5-648b65ebe62f", "id": 1}] 2024-03-29T11:12:02.927 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.948+0000 7fb5257fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:02.927 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-03-29T11:12:02.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.949+0000 7fb52f0e6640 10 librados: watch_flush enter 2024-03-29T11:12:02.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.949+0000 7fb52f0e6640 10 librados: watch_flush exit 2024-03-29T11:12:02.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.949+0000 7fb52f0e6640 20 client.4111.objecter shutdown clearing up homeless session... 2024-03-29T11:12:02.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.949+0000 7fb52f0e6640 10 client.4111.objecter successfully canceled tick 2024-03-29T11:12:02.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.949+0000 7fb52f0e6640 10 monclient: shutdown 2024-03-29T11:12:02.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.949+0000 7fb52f0e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:02.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.949+0000 7fb52f0e6640 1 -- 172.21.15.184:0/3993934927 >> v1:172.21.15.184:6789/0 conn(0x7fb5280b0ad0 legacy=0x7fb5280adab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:02.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.949+0000 7fb52f0e6640 1 -- 172.21.15.184:0/3993934927 shutdown_connections 2024-03-29T11:12:02.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.949+0000 7fb52f0e6640 1 -- 172.21.15.184:0/3993934927 >> 172.21.15.184:0/3993934927 conn(0x7fb5280b4970 msgr2=0x7fb5280b4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:02.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.949+0000 7fb52f0e6640 1 -- 172.21.15.184:0/3993934927 shutdown_connections 2024-03-29T11:12:02.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.949+0000 7fb52f0e6640 1 -- 172.21.15.184:0/3993934927 wait complete. 2024-03-29T11:12:02.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:02.949+0000 7fb52f0e6640 1 librados: shutdown 2024-03-29T11:12:02.938 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new dbf4e71c-275e-4f4e-9f0e-dbdd355ac904 2 2024-03-29T11:12:03.040 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.061+0000 7fb5a582a640 10 monclient: get_monmap_and_config 2024-03-29T11:12:03.040 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.061+0000 7fb5a582a640 10 monclient: build_initial_monmap 2024-03-29T11:12:03.040 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.061+0000 7fb5a582a640 10 monclient: monmap: 2024-03-29T11:12:03.040 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2024-03-29T11:12:03.040 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.040 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:12:03.063119+0000 2024-03-29T11:12:03.040 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:12:03.063119+0000 2024-03-29T11:12:03.040 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:03.040 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.040 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:03.040 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:12:03.041 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:12:03.041 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.049 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.062+0000 7fb5a582a640 1 Processor -- start 2024-03-29T11:12:03.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.062+0000 7fb5a582a640 1 -- start start 2024-03-29T11:12:03.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.062+0000 7fb5a582a640 10 monclient: init 2024-03-29T11:12:03.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.062+0000 7fb5a582a640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:03.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.062+0000 7fb5a582a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:03.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.062+0000 7fb5a582a640 10 monclient(hunting): picked mon.noname-c con 0x7fb5a00743e0 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:03.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.062+0000 7fb5a582a640 10 monclient(hunting): picked mon.noname-a con 0x7fb5a00747c0 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:03.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb5a582a640 10 monclient(hunting): picked mon.noname-b con 0x7fb5a0071bf0 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:03.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb5a582a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a0070bb0 con 0x7fb5a00747c0 2024-03-29T11:12:03.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb5a582a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a0071260 con 0x7fb5a00743e0 2024-03-29T11:12:03.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb5a582a640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a0070570 con 0x7fb5a0071bf0 2024-03-29T11:12:03.051 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb5a582a640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:03.051 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb5a582a640 10 monclient(hunting): authenticate will time out at 913.247047s 2024-03-29T11:12:03.051 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59effd640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fb5a00743e0 0x7fb5a00adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:59492/0 (socket says 172.21.15.184:59492) 2024-03-29T11:12:03.051 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59effd640 1 -- 172.21.15.184:0/1055895176 learned_addr learned my addr 172.21.15.184:0/1055895176 (peer_addr_for_me v1:172.21.15.184:0/0) 2024-03-29T11:12:03.051 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 1 -- 172.21.15.184:0/1055895176 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022088751 0 0) 0x7fb5a0071260 con 0x7fb5a00743e0 2024-03-29T11:12:03.051 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.051 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.051 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.051 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:12:03.051 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 1 -- 172.21.15.184:0/1055895176 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb588003520 con 0x7fb5a00743e0 2024-03-29T11:12:03.051 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 1 -- 172.21.15.184:0/1055895176 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950945899 0 0) 0x7fb5a0070bb0 con 0x7fb5a00747c0 2024-03-29T11:12:03.051 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.051 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.051 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.052 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:12:03.052 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 1 -- 172.21.15.184:0/1055895176 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5a0071260 con 0x7fb5a00747c0 2024-03-29T11:12:03.052 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 1 -- 172.21.15.184:0/1055895176 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910475123 0 0) 0x7fb5a0070570 con 0x7fb5a0071bf0 2024-03-29T11:12:03.052 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.052 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.052 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.052 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:12:03.052 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 1 -- 172.21.15.184:0/1055895176 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5a0070bb0 con 0x7fb5a0071bf0 2024-03-29T11:12:03.052 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 1 -- 172.21.15.184:0/1055895176 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3761505777 0 0) 0x7fb588003520 con 0x7fb5a00743e0 2024-03-29T11:12:03.052 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 1 -- 172.21.15.184:0/1055895176 >> v1:172.21.15.170:6790/0 conn(0x7fb5a0071bf0 legacy=0x7fb5a014b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.052 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 1 -- 172.21.15.184:0/1055895176 >> v1:172.21.15.170:6789/0 conn(0x7fb5a00747c0 legacy=0x7fb5a01433b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.052 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:12:03.052 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 1 monclient: found mon.noname-c 2024-03-29T11:12:03.052 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:03.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 1 -- 172.21.15.184:0/1055895176 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a0070700 con 0x7fb5a00743e0 2024-03-29T11:12:03.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient: _finish_auth 0 2024-03-29T11:12:03.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:12:03.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:03.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb5a582a640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:12:03.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 1 -- 172.21.15.184:0/1055895176 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb594001690 con 0x7fb5a00743e0 2024-03-29T11:12:03.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb5a582a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:03.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:03.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient: dump: 2024-03-29T11:12:03.053 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.053 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.054 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.054 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.054 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.054 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.054 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.054 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.054 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.054 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb5a582a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:03.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 1 -- 172.21.15.184:0/1055895176 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb594001870 con 0x7fb5a00743e0 2024-03-29T11:12:03.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:03.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb5a582a640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:03.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb5a582a640 4 set_mon_vals no callback set 2024-03-29T11:12:03.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 1 -- 172.21.15.184:0/1055895176 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb594001c80 con 0x7fb5a00743e0 2024-03-29T11:12:03.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:03.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.063+0000 7fb59dffb640 10 monclient: dump: 2024-03-29T11:12:03.055 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.055 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.055 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.055 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.055 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.055 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.055 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.055 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.055 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.055 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.064+0000 7fb5a582a640 10 monclient: shutdown 2024-03-29T11:12:03.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.064+0000 7fb5a582a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:03.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.064+0000 7fb5a582a640 1 -- 172.21.15.184:0/1055895176 >> v1:172.21.15.184:6789/0 conn(0x7fb5a00743e0 legacy=0x7fb5a00adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.064+0000 7fb5a582a640 1 -- 172.21.15.184:0/1055895176 shutdown_connections 2024-03-29T11:12:03.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.064+0000 7fb5a582a640 1 -- 172.21.15.184:0/1055895176 >> 172.21.15.184:0/1055895176 conn(0x7fb5a00b4470 msgr2=0x7fb5a00b4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:03.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.064+0000 7fb5a582a640 1 -- 172.21.15.184:0/1055895176 shutdown_connections 2024-03-29T11:12:03.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.064+0000 7fb5a582a640 1 -- 172.21.15.184:0/1055895176 wait complete. 2024-03-29T11:12:03.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.064+0000 7fb5a582a640 10 monclient: build_initial_monmap 2024-03-29T11:12:03.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.064+0000 7fb5a582a640 10 monclient: monmap: 2024-03-29T11:12:03.056 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2024-03-29T11:12:03.056 INFO:teuthology.orchestra.run.smithi184.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:03.056 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 0.000000 2024-03-29T11:12:03.056 INFO:teuthology.orchestra.run.smithi184.stderr:created 0.000000 2024-03-29T11:12:03.056 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:03.056 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.057 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:03.057 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:12:03.057 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:12:03.057 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.065+0000 7fb5a582a640 1 librados: starting msgr at 2024-03-29T11:12:03.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.065+0000 7fb5a582a640 1 librados: starting objecter 2024-03-29T11:12:03.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.065+0000 7fb5a582a640 1 Processor -- start 2024-03-29T11:12:03.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.065+0000 7fb5a582a640 1 -- start start 2024-03-29T11:12:03.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.065+0000 7fb5a582a640 1 librados: setting wanted keys 2024-03-29T11:12:03.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.065+0000 7fb5a582a640 1 librados: calling monclient init 2024-03-29T11:12:03.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.065+0000 7fb5a582a640 10 monclient: init 2024-03-29T11:12:03.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.065+0000 7fb5a582a640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:03.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.065+0000 7fb5a582a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:12:03.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.065+0000 7fb5a582a640 10 monclient(hunting): picked mon.noname-b con 0x7fb5a0071bf0 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:03.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb5a582a640 10 monclient(hunting): picked mon.noname-c con 0x7fb5a00743e0 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:03.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb5a582a640 10 monclient(hunting): picked mon.noname-a con 0x7fb5a013e340 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:03.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb5a582a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a00a9440 con 0x7fb5a013e340 2024-03-29T11:12:03.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb5a582a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a00aa2e0 con 0x7fb5a0071bf0 2024-03-29T11:12:03.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb5a582a640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a0131bd0 con 0x7fb5a00743e0 2024-03-29T11:12:03.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb5a582a640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:03.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb5a582a640 10 monclient(hunting): authenticate will time out at 913.250047s 2024-03-29T11:12:03.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb59effd640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fb5a0071bf0 0x7fb5a013dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:59494/0 (socket says 172.21.15.184:59494) 2024-03-29T11:12:03.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb59effd640 1 -- 172.21.15.184:0/712065813 learned_addr learned my addr 172.21.15.184:0/712065813 (peer_addr_for_me v1:172.21.15.184:0/0) 2024-03-29T11:12:03.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 client.?.objecter ms_handle_connect 0x7fb5a0071bf0 2024-03-29T11:12:03.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:03.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 client.?.objecter ms_handle_connect 0x7fb5a00743e0 2024-03-29T11:12:03.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:03.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 client.?.objecter ms_handle_connect 0x7fb5a013e340 2024-03-29T11:12:03.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:03.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359063688 0 0) 0x7fb5a00aa2e0 con 0x7fb5a0071bf0 2024-03-29T11:12:03.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:12:03.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb58c004eb0 con 0x7fb5a0071bf0 2024-03-29T11:12:03.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167572168 0 0) 0x7fb5a0131bd0 con 0x7fb5a00743e0 2024-03-29T11:12:03.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:12:03.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5a00aa2e0 con 0x7fb5a00743e0 2024-03-29T11:12:03.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2166283959 0 0) 0x7fb5a00a9440 con 0x7fb5a013e340 2024-03-29T11:12:03.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.060 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.060 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:12:03.060 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5a0131bd0 con 0x7fb5a013e340 2024-03-29T11:12:03.060 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4090572993 0 0) 0x7fb5a0131bd0 con 0x7fb5a013e340 2024-03-29T11:12:03.060 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5a00a9440 con 0x7fb5a013e340 2024-03-29T11:12:03.060 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb598001bd0 con 0x7fb5a013e340 2024-03-29T11:12:03.060 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.060 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:03.060 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): dump: 2024-03-29T11:12:03.060 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.060 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.060 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.060 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.060 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.060 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.061 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.061 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.061 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.061 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.061 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:03.061 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 431620746 0 0) 0x7fb58c004eb0 con 0x7fb5a0071bf0 2024-03-29T11:12:03.061 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5a0131bd0 con 0x7fb5a0071bf0 2024-03-29T11:12:03.061 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2699878797 0 0) 0x7fb5a00aa2e0 con 0x7fb5a00743e0 2024-03-29T11:12:03.061 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb58c004eb0 con 0x7fb5a00743e0 2024-03-29T11:12:03.061 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb594001b80 con 0x7fb5a0071bf0 2024-03-29T11:12:03.061 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.061 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:03.061 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): dump: 2024-03-29T11:12:03.062 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.062 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.062 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.062 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.062 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.062 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.062 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.062 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.062 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.062 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.062 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb5900018b0 con 0x7fb5a00743e0 2024-03-29T11:12:03.062 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.062 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:03.062 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient(hunting): dump: 2024-03-29T11:12:03.062 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.063 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.063 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.063 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.063 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.063 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.063 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.063 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.063 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.063 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.063 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1357105166 0 0) 0x7fb5a00a9440 con 0x7fb5a013e340 2024-03-29T11:12:03.063 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 >> v1:172.21.15.170:6790/0 conn(0x7fb5a00743e0 legacy=0x7fb5a0130db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.063 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 >> v1:172.21.15.184:6789/0 conn(0x7fb5a0071bf0 legacy=0x7fb5a013dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.063 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient: _finish_hunting 0 2024-03-29T11:12:03.063 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 monclient: found mon.a 2024-03-29T11:12:03.063 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:03.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:03.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a00aa550 con 0x7fb5a013e340 2024-03-29T11:12:03.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient: _finish_auth 0 2024-03-29T11:12:03.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient: _check_auth_tickets 2024-03-29T11:12:03.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:03.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb5a582a640 5 monclient: authenticate success, global_id 4131 2024-03-29T11:12:03.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb5a582a640 10 monclient: _renew_subs 2024-03-29T11:12:03.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb5a582a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:03.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb5a582a640 1 -- 172.21.15.184:0/712065813 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a00aa830 con 0x7fb5a013e340 2024-03-29T11:12:03.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb5a582a640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:03.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb5a582a640 10 monclient: _renew_subs 2024-03-29T11:12:03.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb5a582a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:03.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb5a582a640 1 -- 172.21.15.184:0/712065813 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a0140f60 con 0x7fb5a013e340 2024-03-29T11:12:03.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb5a582a640 1 librados: init done 2024-03-29T11:12:03.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb598002060 con 0x7fb5a013e340 2024-03-29T11:12:03.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:03.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb598003b30 con 0x7fb5a013e340 2024-03-29T11:12:03.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.066+0000 7fb587fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.067+0000 7fb587fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:03.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.067+0000 7fb59d7fa640 4 set_mon_vals no callback set 2024-03-29T11:12:03.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.067+0000 7fb587fff640 10 monclient: dump: 2024-03-29T11:12:03.065 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.065 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.065 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.065 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.065 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.065 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.065 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.065 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.066 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.066 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.067+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb598004020 con 0x7fb5a013e340 2024-03-29T11:12:03.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.067+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 129774537 0 0) 0x7fb598005630 con 0x7fb5a013e340 2024-03-29T11:12:03.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.067+0000 7fb587fff640 10 client.4131.objecter ms_dispatch 0x7fb5a0000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:12:03.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.067+0000 7fb587fff640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:12:03.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.067+0000 7fb587fff640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:12:03.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.067+0000 7fb587fff640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-29T11:12:03.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.067+0000 7fb5a582a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:03.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.067+0000 7fb5a582a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:03.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.067+0000 7fb5a582a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:03.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.067+0000 7fb5a582a640 1 -- 172.21.15.184:0/712065813 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb580005080 con 0x7fb5a013e340 2024-03-29T11:12:03.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.069+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb5980052c0 con 0x7fb5a013e340 2024-03-29T11:12:03.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.069+0000 7fb587fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:03.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.069+0000 7fb587fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:03.067 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:03.076+0000 7f84d7221040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:12:03.107 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:03.129+0000 7f84d7221040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:12:03.113 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:03.135+0000 7f1e7ea43040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:12:03.120 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.142+0000 7fb5a582a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dbf4e71c-275e-4f4e-9f0e-dbdd355ac904", "id": 2}] 2024-03-29T11:12:03.120 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.142+0000 7fb5a582a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dbf4e71c-275e-4f4e-9f0e-dbdd355ac904", "id": 2}] 2024-03-29T11:12:03.120 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.142+0000 7fb5a582a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:03.120 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.142+0000 7fb5a582a640 1 -- 172.21.15.184:0/712065813 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd new", "uuid": "dbf4e71c-275e-4f4e-9f0e-dbdd355ac904", "id": 2} v 0) v1 -- 0x7fb5800058b0 con 0x7fb5a013e340 2024-03-29T11:12:03.121 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.143+0000 7fb587fff640 1 -- 172.21.15.184:0/712065813 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dbf4e71c-275e-4f4e-9f0e-dbdd355ac904", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 932727631 0 216599125) 0x7fb5980058c0 con 0x7fb5a013e340 2024-03-29T11:12:03.121 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.143+0000 7fb587fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dbf4e71c-275e-4f4e-9f0e-dbdd355ac904", "id": 2}] 2024-03-29T11:12:03.121 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.143+0000 7fb587fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:03.122 INFO:teuthology.orchestra.run.smithi184.stdout:2 2024-03-29T11:12:03.123 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.144+0000 7fb5a582a640 10 librados: watch_flush enter 2024-03-29T11:12:03.123 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.144+0000 7fb5a582a640 10 librados: watch_flush exit 2024-03-29T11:12:03.123 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.144+0000 7fb5a582a640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-29T11:12:03.123 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.144+0000 7fb5a582a640 10 client.4131.objecter successfully canceled tick 2024-03-29T11:12:03.123 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.144+0000 7fb5a582a640 10 monclient: shutdown 2024-03-29T11:12:03.123 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.144+0000 7fb5a582a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:03.123 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.144+0000 7fb5a582a640 1 -- 172.21.15.184:0/712065813 >> v1:172.21.15.170:6789/0 conn(0x7fb5a013e340 legacy=0x7fb5a01314c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.123 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.144+0000 7fb5a582a640 1 -- 172.21.15.184:0/712065813 shutdown_connections 2024-03-29T11:12:03.123 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.144+0000 7fb5a582a640 1 -- 172.21.15.184:0/712065813 >> 172.21.15.184:0/712065813 conn(0x7fb5a00b4470 msgr2=0x7fb5a00b1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:03.123 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.144+0000 7fb5a582a640 1 -- 172.21.15.184:0/712065813 shutdown_connections 2024-03-29T11:12:03.124 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.144+0000 7fb5a582a640 1 -- 172.21.15.184:0/712065813 wait complete. 2024-03-29T11:12:03.124 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.144+0000 7fb5a582a640 1 librados: shutdown 2024-03-29T11:12:03.133 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 12e77d37-9ec0-49b7-acec-0a241a418fce 3 2024-03-29T11:12:03.151 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:03.174+0000 7f84d7221040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:12:03.172 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:03.194+0000 7f1e7ea43040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:12:03.204 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:03.227+0000 7f84d7221040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:12:03.220 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:03.242+0000 7f1e7ea43040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:12:03.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.254+0000 7f4c65be8640 10 monclient: get_monmap_and_config 2024-03-29T11:12:03.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.254+0000 7f4c65be8640 10 monclient: build_initial_monmap 2024-03-29T11:12:03.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.254+0000 7f4c65be8640 10 monclient: monmap: 2024-03-29T11:12:03.233 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2024-03-29T11:12:03.233 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.233 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:12:03.256075+0000 2024-03-29T11:12:03.233 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:12:03.256075+0000 2024-03-29T11:12:03.233 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:03.233 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.233 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:03.233 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:12:03.233 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:12:03.234 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.244 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.255+0000 7f4c65be8640 1 Processor -- start 2024-03-29T11:12:03.244 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.255+0000 7f4c65be8640 1 -- start start 2024-03-29T11:12:03.244 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.255+0000 7f4c65be8640 10 monclient: init 2024-03-29T11:12:03.244 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c65be8640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:03.244 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c65be8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:03.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c65be8640 10 monclient(hunting): picked mon.noname-c con 0x7f4c6009ac40 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:03.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c65be8640 10 monclient(hunting): picked mon.noname-a con 0x7f4c6009c870 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:03.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c65be8640 10 monclient(hunting): picked mon.noname-b con 0x7f4c600b4360 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:03.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c65be8640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6009dec0 con 0x7f4c6009c870 2024-03-29T11:12:03.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c65be8640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6009da30 con 0x7f4c6009ac40 2024-03-29T11:12:03.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c65be8640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c600b39c0 con 0x7f4c600b4360 2024-03-29T11:12:03.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c65be8640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:03.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c65be8640 10 monclient(hunting): authenticate will time out at 913.440228s 2024-03-29T11:12:03.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5f7fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f4c6009ac40 0x7f4c600b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:59510/0 (socket says 172.21.15.184:59510) 2024-03-29T11:12:03.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5f7fe640 1 -- 172.21.15.184:0/1492912528 learned_addr learned my addr 172.21.15.184:0/1492912528 (peer_addr_for_me v1:172.21.15.184:0/0) 2024-03-29T11:12:03.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 1 -- 172.21.15.184:0/1492912528 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242415232 0 0) 0x7f4c6009da30 con 0x7f4c6009ac40 2024-03-29T11:12:03.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:12:03.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 1 -- 172.21.15.184:0/1492912528 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c4c003520 con 0x7f4c6009ac40 2024-03-29T11:12:03.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 1 -- 172.21.15.184:0/1492912528 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865755706 0 0) 0x7f4c600b39c0 con 0x7f4c600b4360 2024-03-29T11:12:03.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:12:03.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 1 -- 172.21.15.184:0/1492912528 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c6009da30 con 0x7f4c600b4360 2024-03-29T11:12:03.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 1 -- 172.21.15.184:0/1492912528 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675308724 0 0) 0x7f4c48001110 con 0x7f4c6009c870 2024-03-29T11:12:03.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:12:03.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 1 -- 172.21.15.184:0/1492912528 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c600b39c0 con 0x7f4c6009c870 2024-03-29T11:12:03.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 1 -- 172.21.15.184:0/1492912528 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 278987239 0 0) 0x7f4c4c003520 con 0x7f4c6009ac40 2024-03-29T11:12:03.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 1 -- 172.21.15.184:0/1492912528 >> v1:172.21.15.170:6790/0 conn(0x7f4c600b4360 legacy=0x7f4c6014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 1 -- 172.21.15.184:0/1492912528 >> v1:172.21.15.170:6789/0 conn(0x7f4c6009c870 legacy=0x7f4c6013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:12:03.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 1 monclient: found mon.noname-c 2024-03-29T11:12:03.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:03.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 1 -- 172.21.15.184:0/1492912528 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c600b5160 con 0x7f4c6009ac40 2024-03-29T11:12:03.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:12:03.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:12:03.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:03.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 1 -- 172.21.15.184:0/1492912528 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f4c50001690 con 0x7f4c6009ac40 2024-03-29T11:12:03.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:03.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient: dump: 2024-03-29T11:12:03.248 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.248 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.248 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.248 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.248 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.248 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.248 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.248 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.249 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.249 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.249 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c65be8640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:12:03.249 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c65be8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:03.249 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 1 -- 172.21.15.184:0/1492912528 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c50001870 con 0x7f4c6009ac40 2024-03-29T11:12:03.249 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:03.249 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 1 -- 172.21.15.184:0/1492912528 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f4c50001c80 con 0x7f4c6009ac40 2024-03-29T11:12:03.249 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c65be8640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:03.249 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c65be8640 4 set_mon_vals no callback set 2024-03-29T11:12:03.249 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.249 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:03.249 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.256+0000 7f4c5e7fc640 10 monclient: dump: 2024-03-29T11:12:03.249 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.249 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.250 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.250 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.250 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.250 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.250 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.250 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.250 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.250 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.257+0000 7f4c65be8640 10 monclient: shutdown 2024-03-29T11:12:03.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.257+0000 7f4c65be8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:03.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.257+0000 7f4c65be8640 1 -- 172.21.15.184:0/1492912528 >> v1:172.21.15.184:6789/0 conn(0x7f4c6009ac40 legacy=0x7f4c600b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.257+0000 7f4c65be8640 1 -- 172.21.15.184:0/1492912528 shutdown_connections 2024-03-29T11:12:03.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.257+0000 7f4c65be8640 1 -- 172.21.15.184:0/1492912528 >> 172.21.15.184:0/1492912528 conn(0x7f4c60066b00 msgr2=0x7f4c60066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:03.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.257+0000 7f4c65be8640 1 -- 172.21.15.184:0/1492912528 shutdown_connections 2024-03-29T11:12:03.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.257+0000 7f4c65be8640 1 -- 172.21.15.184:0/1492912528 wait complete. 2024-03-29T11:12:03.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.258+0000 7f4c65be8640 10 monclient: build_initial_monmap 2024-03-29T11:12:03.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.258+0000 7f4c65be8640 10 monclient: monmap: 2024-03-29T11:12:03.251 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2024-03-29T11:12:03.251 INFO:teuthology.orchestra.run.smithi184.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:03.251 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 0.000000 2024-03-29T11:12:03.251 INFO:teuthology.orchestra.run.smithi184.stderr:created 0.000000 2024-03-29T11:12:03.251 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:03.251 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.251 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:03.251 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:12:03.251 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:12:03.251 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.258+0000 7f4c65be8640 1 librados: starting msgr at 2024-03-29T11:12:03.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.258+0000 7f4c65be8640 1 librados: starting objecter 2024-03-29T11:12:03.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.258+0000 7f4c65be8640 1 Processor -- start 2024-03-29T11:12:03.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.258+0000 7f4c65be8640 1 -- start start 2024-03-29T11:12:03.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.258+0000 7f4c65be8640 1 librados: setting wanted keys 2024-03-29T11:12:03.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.258+0000 7f4c65be8640 1 librados: calling monclient init 2024-03-29T11:12:03.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.258+0000 7f4c65be8640 10 monclient: init 2024-03-29T11:12:03.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c65be8640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:03.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c65be8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:03.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c65be8640 10 monclient(hunting): picked mon.noname-a con 0x7f4c6009ac40 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:03.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c65be8640 10 monclient(hunting): picked mon.noname-c con 0x7f4c60138f60 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:03.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c65be8640 10 monclient(hunting): picked mon.noname-b con 0x7f4c600afdc0 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:03.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c65be8640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c601db610 con 0x7f4c6009ac40 2024-03-29T11:12:03.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c65be8640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c60139a00 con 0x7f4c600afdc0 2024-03-29T11:12:03.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c65be8640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c600b0230 con 0x7f4c60138f60 2024-03-29T11:12:03.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c65be8640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:03.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c65be8640 10 monclient(hunting): authenticate will time out at 913.443235s 2024-03-29T11:12:03.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c5ffff640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f4c600afdc0 0x7f4c601e1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:59520/0 (socket says 172.21.15.184:59520) 2024-03-29T11:12:03.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c5ffff640 1 -- 172.21.15.184:0/2089149932 learned_addr learned my addr 172.21.15.184:0/2089149932 (peer_addr_for_me v1:172.21.15.184:0/0) 2024-03-29T11:12:03.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 client.?.objecter ms_handle_connect 0x7f4c600afdc0 2024-03-29T11:12:03.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:03.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 client.?.objecter ms_handle_connect 0x7f4c60138f60 2024-03-29T11:12:03.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:03.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 client.?.objecter ms_handle_connect 0x7f4c6009ac40 2024-03-29T11:12:03.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:03.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172698058 0 0) 0x7f4c60139a00 con 0x7f4c600afdc0 2024-03-29T11:12:03.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:12:03.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c40004eb0 con 0x7f4c600afdc0 2024-03-29T11:12:03.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074254665 0 0) 0x7f4c600b0230 con 0x7f4c60138f60 2024-03-29T11:12:03.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:12:03.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c60139a00 con 0x7f4c60138f60 2024-03-29T11:12:03.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488876257 0 0) 0x7f4c601db610 con 0x7f4c6009ac40 2024-03-29T11:12:03.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:12:03.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c600b0230 con 0x7f4c6009ac40 2024-03-29T11:12:03.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 557586158 0 0) 0x7f4c40004eb0 con 0x7f4c600afdc0 2024-03-29T11:12:03.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c601db610 con 0x7f4c600afdc0 2024-03-29T11:12:03.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f4c58001c00 con 0x7f4c600afdc0 2024-03-29T11:12:03.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:03.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): dump: 2024-03-29T11:12:03.255 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.255 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.255 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.255 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.255 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.256 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.256 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.256 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.256 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.256 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:03.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 978533628 0 0) 0x7f4c60139a00 con 0x7f4c60138f60 2024-03-29T11:12:03.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c40004eb0 con 0x7f4c60138f60 2024-03-29T11:12:03.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f4c48001860 con 0x7f4c60138f60 2024-03-29T11:12:03.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:03.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): dump: 2024-03-29T11:12:03.256 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.256 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.256 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.256 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.257 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.257 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.257 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.257 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.257 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.257 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1779974014 0 0) 0x7f4c600b0230 con 0x7f4c6009ac40 2024-03-29T11:12:03.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c60139a00 con 0x7f4c6009ac40 2024-03-29T11:12:03.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f4c500019e0 con 0x7f4c6009ac40 2024-03-29T11:12:03.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:03.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient(hunting): dump: 2024-03-29T11:12:03.257 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.257 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.258 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.258 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.258 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.258 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.258 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.258 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.258 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.258 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1522135790 0 0) 0x7f4c601db610 con 0x7f4c600afdc0 2024-03-29T11:12:03.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 >> v1:172.21.15.170:6790/0 conn(0x7f4c60138f60 legacy=0x7f4c600af6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 >> v1:172.21.15.170:6789/0 conn(0x7f4c6009ac40 legacy=0x7f4c601dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient: _finish_hunting 0 2024-03-29T11:12:03.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 1 monclient: found mon.b 2024-03-29T11:12:03.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:03.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.259+0000 7f4c57fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c600b03c0 con 0x7f4c600afdc0 2024-03-29T11:12:03.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c57fff640 10 monclient: _finish_auth 0 2024-03-29T11:12:03.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c57fff640 10 monclient: _check_auth_tickets 2024-03-29T11:12:03.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:03.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c65be8640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:12:03.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c65be8640 10 monclient: _renew_subs 2024-03-29T11:12:03.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c65be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c65be8640 1 -- 172.21.15.184:0/2089149932 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c601e36e0 con 0x7f4c600afdc0 2024-03-29T11:12:03.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c65be8640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:03.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c65be8640 10 monclient: _renew_subs 2024-03-29T11:12:03.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c65be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c65be8640 1 -- 172.21.15.184:0/2089149932 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c601e3a50 con 0x7f4c600afdc0 2024-03-29T11:12:03.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c65be8640 1 librados: init done 2024-03-29T11:12:03.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c580020d0 con 0x7f4c600afdc0 2024-03-29T11:12:03.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c57fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:03.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f4c58003990 con 0x7f4c600afdc0 2024-03-29T11:12:03.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c5dffb640 4 set_mon_vals no callback set 2024-03-29T11:12:03.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c57fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:03.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c57fff640 10 monclient: dump: 2024-03-29T11:12:03.261 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.261 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.261 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.261 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.261 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.261 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.261 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.261 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.261 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.261 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.261 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4c58003f00 con 0x7f4c600afdc0 2024-03-29T11:12:03.261 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2463141847 0 0) 0x7f4c58005b40 con 0x7f4c600afdc0 2024-03-29T11:12:03.261 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c57fff640 10 client.4123.objecter ms_dispatch 0x7f4c60000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:12:03.262 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c57fff640 3 client.4123.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:12:03.262 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c57fff640 3 client.4123.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:12:03.262 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.260+0000 7f4c57fff640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T11:12:03.262 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.261+0000 7f4c65be8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:03.262 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.261+0000 7f4c65be8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:03.262 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.261+0000 7f4c65be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.262 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.261+0000 7f4c65be8640 1 -- 172.21.15.184:0/2089149932 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c44005080 con 0x7f4c600afdc0 2024-03-29T11:12:03.262 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.263+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4c580041b0 con 0x7f4c600afdc0 2024-03-29T11:12:03.262 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.263+0000 7f4c57fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:03.262 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.263+0000 7f4c57fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:03.263 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:03.269+0000 7f84d7221040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:12:03.287 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:03.308+0000 7f1e7ea43040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:12:03.325 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.346+0000 7f4c65be8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "12e77d37-9ec0-49b7-acec-0a241a418fce", "id": 3}] 2024-03-29T11:12:03.325 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.346+0000 7f4c65be8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "12e77d37-9ec0-49b7-acec-0a241a418fce", "id": 3}] 2024-03-29T11:12:03.325 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.346+0000 7f4c65be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.325 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.346+0000 7f4c65be8640 1 -- 172.21.15.184:0/2089149932 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd new", "uuid": "12e77d37-9ec0-49b7-acec-0a241a418fce", "id": 3} v 0) v1 -- 0x7f4c440058b0 con 0x7f4c600afdc0 2024-03-29T11:12:03.328 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.350+0000 7f4c57fff640 1 -- 172.21.15.184:0/2089149932 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "12e77d37-9ec0-49b7-acec-0a241a418fce", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2356392286 0 525046306) 0x7f4c580044a0 con 0x7f4c600afdc0 2024-03-29T11:12:03.329 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.350+0000 7f4c57fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "12e77d37-9ec0-49b7-acec-0a241a418fce", "id": 3}] 2024-03-29T11:12:03.329 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.350+0000 7f4c57fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:03.329 INFO:teuthology.orchestra.run.smithi184.stdout:3 2024-03-29T11:12:03.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.351+0000 7f4c65be8640 10 librados: watch_flush enter 2024-03-29T11:12:03.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.351+0000 7f4c65be8640 10 librados: watch_flush exit 2024-03-29T11:12:03.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.351+0000 7f4c65be8640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T11:12:03.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.351+0000 7f4c65be8640 10 client.4123.objecter successfully canceled tick 2024-03-29T11:12:03.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.351+0000 7f4c65be8640 10 monclient: shutdown 2024-03-29T11:12:03.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.351+0000 7f4c65be8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:03.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.351+0000 7f4c65be8640 1 -- 172.21.15.184:0/2089149932 >> v1:172.21.15.184:6789/0 conn(0x7f4c600afdc0 legacy=0x7f4c601e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.351+0000 7f4c65be8640 1 -- 172.21.15.184:0/2089149932 shutdown_connections 2024-03-29T11:12:03.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.351+0000 7f4c65be8640 1 -- 172.21.15.184:0/2089149932 >> 172.21.15.184:0/2089149932 conn(0x7f4c60066b00 msgr2=0x7f4c600b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:03.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.351+0000 7f4c65be8640 1 -- 172.21.15.184:0/2089149932 shutdown_connections 2024-03-29T11:12:03.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.351+0000 7f4c65be8640 1 -- 172.21.15.184:0/2089149932 wait complete. 2024-03-29T11:12:03.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.351+0000 7f4c65be8640 1 librados: shutdown 2024-03-29T11:12:03.331 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:03.352+0000 7f1e7ea43040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:12:03.340 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 12a0b442-15be-4896-b121-042507f91881 4 2024-03-29T11:12:03.361 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:03.384+0000 7f84d7221040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:12:03.431 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:03.454+0000 7f84d7221040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:12:03.452 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:03.474+0000 7f1e7ea43040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:12:03.476 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:03.499+0000 7f84d7221040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:12:03.491 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.502+0000 7f4f56814640 10 monclient: get_monmap_and_config 2024-03-29T11:12:03.491 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.502+0000 7f4f56814640 10 monclient: build_initial_monmap 2024-03-29T11:12:03.491 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.502+0000 7f4f56814640 10 monclient: monmap: 2024-03-29T11:12:03.491 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2024-03-29T11:12:03.492 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.492 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:12:03.503370+0000 2024-03-29T11:12:03.492 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:12:03.503370+0000 2024-03-29T11:12:03.492 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:03.492 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.492 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:03.492 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:12:03.492 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:12:03.492 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.492 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.502+0000 7f4f56814640 1 Processor -- start 2024-03-29T11:12:03.492 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.502+0000 7f4f56814640 1 -- start start 2024-03-29T11:12:03.492 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.502+0000 7f4f56814640 10 monclient: init 2024-03-29T11:12:03.492 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f56814640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:03.492 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f56814640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:12:03.492 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f56814640 10 monclient(hunting): picked mon.noname-b con 0x7f4f5009ac40 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:03.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f56814640 10 monclient(hunting): picked mon.noname-a con 0x7f4f5009c870 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:03.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f56814640 10 monclient(hunting): picked mon.noname-c con 0x7f4f500b4360 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:03.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f56814640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f5009dec0 con 0x7f4f5009c870 2024-03-29T11:12:03.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f56814640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f5009da30 con 0x7f4f500b4360 2024-03-29T11:12:03.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f56814640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f500b39c0 con 0x7f4f5009ac40 2024-03-29T11:12:03.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f56814640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:03.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f56814640 10 monclient(hunting): authenticate will time out at 913.687444s 2024-03-29T11:12:03.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f54db1640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f4f500b4360 0x7f4f5014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:59536/0 (socket says 172.21.15.184:59536) 2024-03-29T11:12:03.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f54db1640 1 -- 172.21.15.184:0/1517956856 learned_addr learned my addr 172.21.15.184:0/1517956856 (peer_addr_for_me v1:172.21.15.184:0/0) 2024-03-29T11:12:03.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f4effd640 1 -- 172.21.15.184:0/1517956856 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270199768 0 0) 0x7f4f5009da30 con 0x7f4f500b4360 2024-03-29T11:12:03.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f4effd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f4effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f4effd640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:12:03.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f4effd640 1 -- 172.21.15.184:0/1517956856 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f38003520 con 0x7f4f500b4360 2024-03-29T11:12:03.494 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f4effd640 1 -- 172.21.15.184:0/1517956856 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397057698 0 0) 0x7f4f500b39c0 con 0x7f4f5009ac40 2024-03-29T11:12:03.494 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f4effd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.494 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f4effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.494 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.494 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f4effd640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:12:03.494 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f4effd640 1 -- 172.21.15.184:0/1517956856 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f5009da30 con 0x7f4f5009ac40 2024-03-29T11:12:03.494 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f4effd640 1 -- 172.21.15.184:0/1517956856 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458272258 0 0) 0x7f4f40001110 con 0x7f4f5009c870 2024-03-29T11:12:03.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f4effd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f4effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f4effd640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:12:03.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.503+0000 7f4f4effd640 1 -- 172.21.15.184:0/1517956856 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f500b39c0 con 0x7f4f5009c870 2024-03-29T11:12:03.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 1 -- 172.21.15.184:0/1517956856 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2860279787 0 0) 0x7f4f38003520 con 0x7f4f500b4360 2024-03-29T11:12:03.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 1 -- 172.21.15.184:0/1517956856 >> v1:172.21.15.170:6790/0 conn(0x7f4f5009ac40 legacy=0x7f4f500b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 1 -- 172.21.15.184:0/1517956856 >> v1:172.21.15.170:6789/0 conn(0x7f4f5009c870 legacy=0x7f4f5013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:03.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 1 monclient: found mon.noname-c 2024-03-29T11:12:03.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:03.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 1 -- 172.21.15.184:0/1517956856 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f500b5160 con 0x7f4f500b4360 2024-03-29T11:12:03.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 10 monclient: _finish_auth 0 2024-03-29T11:12:03.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:03.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:03.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f56814640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:12:03.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 1 -- 172.21.15.184:0/1517956856 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f4f48001a10 con 0x7f4f500b4360 2024-03-29T11:12:03.497 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.497 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:03.497 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 10 monclient: dump: 2024-03-29T11:12:03.498 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.498 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.498 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.498 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.498 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.498 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.498 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.498 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.498 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.498 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.498 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f56814640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:03.498 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 1 -- 172.21.15.184:0/1517956856 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f48001bf0 con 0x7f4f500b4360 2024-03-29T11:12:03.498 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:03.498 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f56814640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:03.499 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 1 -- 172.21.15.184:0/1517956856 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f4f48002000 con 0x7f4f500b4360 2024-03-29T11:12:03.499 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f56814640 4 set_mon_vals no callback set 2024-03-29T11:12:03.499 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.499 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:03.499 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f4effd640 10 monclient: dump: 2024-03-29T11:12:03.499 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.499 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.499 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.499 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.499 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.500 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.500 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.500 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.500 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.500 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.500 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f56814640 10 monclient: shutdown 2024-03-29T11:12:03.500 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f56814640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:03.500 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f56814640 1 -- 172.21.15.184:0/1517956856 >> v1:172.21.15.184:6789/0 conn(0x7f4f500b4360 legacy=0x7f4f5014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.500 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f56814640 1 -- 172.21.15.184:0/1517956856 shutdown_connections 2024-03-29T11:12:03.500 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f56814640 1 -- 172.21.15.184:0/1517956856 >> 172.21.15.184:0/1517956856 conn(0x7f4f50066b00 msgr2=0x7f4f50066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:03.501 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f56814640 1 -- 172.21.15.184:0/1517956856 shutdown_connections 2024-03-29T11:12:03.501 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.504+0000 7f4f56814640 1 -- 172.21.15.184:0/1517956856 wait complete. 2024-03-29T11:12:03.501 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.505+0000 7f4f56814640 10 monclient: build_initial_monmap 2024-03-29T11:12:03.501 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.505+0000 7f4f56814640 10 monclient: monmap: 2024-03-29T11:12:03.501 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2024-03-29T11:12:03.501 INFO:teuthology.orchestra.run.smithi184.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:03.501 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 0.000000 2024-03-29T11:12:03.501 INFO:teuthology.orchestra.run.smithi184.stderr:created 0.000000 2024-03-29T11:12:03.501 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:03.501 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.501 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:03.501 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:12:03.501 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:12:03.501 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.505+0000 7f4f56814640 1 librados: starting msgr at 2024-03-29T11:12:03.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.505+0000 7f4f56814640 1 librados: starting objecter 2024-03-29T11:12:03.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.505+0000 7f4f56814640 1 Processor -- start 2024-03-29T11:12:03.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.505+0000 7f4f56814640 1 -- start start 2024-03-29T11:12:03.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.505+0000 7f4f56814640 1 librados: setting wanted keys 2024-03-29T11:12:03.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.505+0000 7f4f56814640 1 librados: calling monclient init 2024-03-29T11:12:03.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.505+0000 7f4f56814640 10 monclient: init 2024-03-29T11:12:03.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f56814640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:03.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f56814640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:03.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f56814640 10 monclient(hunting): picked mon.noname-a con 0x7f4f5009ac40 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:03.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f56814640 10 monclient(hunting): picked mon.noname-b con 0x7f4f500b7310 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:03.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f56814640 10 monclient(hunting): picked mon.noname-c con 0x7f4f501d7de0 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:03.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f56814640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f500b78c0 con 0x7f4f5009ac40 2024-03-29T11:12:03.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f56814640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f500b7d50 con 0x7f4f500b7310 2024-03-29T11:12:03.503 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f56814640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f50146480 con 0x7f4f501d7de0 2024-03-29T11:12:03.503 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f56814640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:03.503 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f56814640 10 monclient(hunting): authenticate will time out at 913.690508s 2024-03-29T11:12:03.503 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4f7fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f4f500b7310 0x7f4f501d76d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:59538/0 (socket says 172.21.15.184:59538) 2024-03-29T11:12:03.503 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4f7fe640 1 -- 172.21.15.184:0/1042342487 learned_addr learned my addr 172.21.15.184:0/1042342487 (peer_addr_for_me v1:172.21.15.184:0/0) 2024-03-29T11:12:03.503 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 10 client.?.objecter ms_handle_connect 0x7f4f500b7310 2024-03-29T11:12:03.503 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:03.503 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 10 client.?.objecter ms_handle_connect 0x7f4f5009ac40 2024-03-29T11:12:03.503 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:03.503 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 10 client.?.objecter ms_handle_connect 0x7f4f501d7de0 2024-03-29T11:12:03.503 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:03.503 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311151009 0 0) 0x7f4f500b7d50 con 0x7f4f500b7310 2024-03-29T11:12:03.503 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.504 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.504 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.504 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:12:03.504 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f3c004eb0 con 0x7f4f500b7310 2024-03-29T11:12:03.504 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466548618 0 0) 0x7f4f500b78c0 con 0x7f4f5009ac40 2024-03-29T11:12:03.504 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.504 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.504 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.504 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:12:03.504 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f500b7d50 con 0x7f4f5009ac40 2024-03-29T11:12:03.504 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902655645 0 0) 0x7f4f50146480 con 0x7f4f501d7de0 2024-03-29T11:12:03.504 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.504 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.504 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.504 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:12:03.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f500b78c0 con 0x7f4f501d7de0 2024-03-29T11:12:03.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.506+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1604681074 0 0) 0x7f4f3c004eb0 con 0x7f4f500b7310 2024-03-29T11:12:03.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f50146480 con 0x7f4f500b7310 2024-03-29T11:12:03.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f4f400017e0 con 0x7f4f500b7310 2024-03-29T11:12:03.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:03.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 10 monclient(hunting): dump: 2024-03-29T11:12:03.505 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.505 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.505 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.505 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.505 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.506 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.506 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.506 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.506 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.506 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.506 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:03.507 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2674129080 0 0) 0x7f4f500b7d50 con 0x7f4f5009ac40 2024-03-29T11:12:03.507 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f3c004eb0 con 0x7f4f5009ac40 2024-03-29T11:12:03.507 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f4f44001920 con 0x7f4f5009ac40 2024-03-29T11:12:03.507 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.507 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:03.507 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 10 monclient(hunting): dump: 2024-03-29T11:12:03.507 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.507 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.507 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.507 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.507 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.507 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.507 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.508 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.508 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.508 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.508 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2254097469 0 0) 0x7f4f500b78c0 con 0x7f4f501d7de0 2024-03-29T11:12:03.508 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f500b7d50 con 0x7f4f501d7de0 2024-03-29T11:12:03.508 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f4f48001d60 con 0x7f4f501d7de0 2024-03-29T11:12:03.508 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.508 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:03.508 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 10 monclient(hunting): dump: 2024-03-29T11:12:03.508 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.508 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.508 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.508 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.508 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.509 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.509 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.509 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.509 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.509 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.509 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1430838641 0 0) 0x7f4f50146480 con 0x7f4f500b7310 2024-03-29T11:12:03.509 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 >> v1:172.21.15.170:6790/0 conn(0x7f4f501d7de0 legacy=0x7f4f501e2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.509 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 >> v1:172.21.15.170:6789/0 conn(0x7f4f5009ac40 legacy=0x7f4f500b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.509 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:12:03.509 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 1 monclient: found mon.b 2024-03-29T11:12:03.509 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:03.509 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.509 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f501e68d0 con 0x7f4f500b7310 2024-03-29T11:12:03.509 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 10 monclient: _finish_auth 0 2024-03-29T11:12:03.510 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:12:03.510 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:03.510 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f56814640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:12:03.510 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f56814640 10 monclient: _renew_subs 2024-03-29T11:12:03.510 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f56814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.510 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f56814640 1 -- 172.21.15.184:0/1042342487 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f501e38a0 con 0x7f4f500b7310 2024-03-29T11:12:03.510 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f56814640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:03.510 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f56814640 10 monclient: _renew_subs 2024-03-29T11:12:03.510 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f56814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.510 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f56814640 1 -- 172.21.15.184:0/1042342487 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f501e3dd0 con 0x7f4f500b7310 2024-03-29T11:12:03.510 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f56814640 1 librados: init done 2024-03-29T11:12:03.510 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f40001cd0 con 0x7f4f500b7310 2024-03-29T11:12:03.510 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:03.510 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f4f400037b0 con 0x7f4f500b7310 2024-03-29T11:12:03.511 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.511 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:03.511 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 10 monclient: dump: 2024-03-29T11:12:03.511 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.511 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.511 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.511 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.511 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.512 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.512 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.512 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.512 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.512 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.512 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4e7fc640 4 set_mon_vals no callback set 2024-03-29T11:12:03.512 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4f40003ca0 con 0x7f4f500b7310 2024-03-29T11:12:03.512 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2897618433 0 0) 0x7f4f40004460 con 0x7f4f500b7310 2024-03-29T11:12:03.512 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 10 client.4129.objecter ms_dispatch 0x7f4f50000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:12:03.513 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 3 client.4129.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:12:03.513 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 3 client.4129.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:12:03.513 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.507+0000 7f4f4cff9640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T11:12:03.513 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.508+0000 7f4f56814640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:03.513 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.508+0000 7f4f56814640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:03.513 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.508+0000 7f4f56814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.513 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.508+0000 7f4f56814640 1 -- 172.21.15.184:0/1042342487 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f30005080 con 0x7f4f500b7310 2024-03-29T11:12:03.513 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.510+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4f400057c0 con 0x7f4f500b7310 2024-03-29T11:12:03.513 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.510+0000 7f4f4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:03.513 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.510+0000 7f4f4cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:03.526 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:03.548+0000 7f1e7ea43040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:12:03.529 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:03.552+0000 7f84d7221040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:12:03.559 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.580+0000 7f4f56814640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "12a0b442-15be-4896-b121-042507f91881", "id": 4}] 2024-03-29T11:12:03.559 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.580+0000 7f4f56814640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "12a0b442-15be-4896-b121-042507f91881", "id": 4}] 2024-03-29T11:12:03.559 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.580+0000 7f4f56814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.559 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.580+0000 7f4f56814640 1 -- 172.21.15.184:0/1042342487 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd new", "uuid": "12a0b442-15be-4896-b121-042507f91881", "id": 4} v 0) v1 -- 0x7f4f300058b0 con 0x7f4f500b7310 2024-03-29T11:12:03.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.584+0000 7f4f4cff9640 1 -- 172.21.15.184:0/1042342487 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "12a0b442-15be-4896-b121-042507f91881", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3008368156 0 1697012583) 0x7f4f40004020 con 0x7f4f500b7310 2024-03-29T11:12:03.563 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.584+0000 7f4f4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "12a0b442-15be-4896-b121-042507f91881", "id": 4}] 2024-03-29T11:12:03.563 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.584+0000 7f4f4cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:03.563 INFO:teuthology.orchestra.run.smithi184.stdout:4 2024-03-29T11:12:03.565 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.585+0000 7f4f56814640 10 librados: watch_flush enter 2024-03-29T11:12:03.565 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.585+0000 7f4f56814640 10 librados: watch_flush exit 2024-03-29T11:12:03.565 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.585+0000 7f4f56814640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T11:12:03.565 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.585+0000 7f4f56814640 10 client.4129.objecter successfully canceled tick 2024-03-29T11:12:03.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.585+0000 7f4f56814640 10 monclient: shutdown 2024-03-29T11:12:03.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.585+0000 7f4f56814640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:03.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.585+0000 7f4f56814640 1 -- 172.21.15.184:0/1042342487 >> v1:172.21.15.184:6789/0 conn(0x7f4f500b7310 legacy=0x7f4f501d76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.585+0000 7f4f56814640 1 -- 172.21.15.184:0/1042342487 shutdown_connections 2024-03-29T11:12:03.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.585+0000 7f4f56814640 1 -- 172.21.15.184:0/1042342487 >> 172.21.15.184:0/1042342487 conn(0x7f4f50066b00 msgr2=0x7f4f5013a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:03.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.585+0000 7f4f56814640 1 -- 172.21.15.184:0/1042342487 shutdown_connections 2024-03-29T11:12:03.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.585+0000 7f4f56814640 1 -- 172.21.15.184:0/1042342487 wait complete. 2024-03-29T11:12:03.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.585+0000 7f4f56814640 1 librados: shutdown 2024-03-29T11:12:03.574 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 9442ffc6-a927-47de-a9f3-7f8cb0ed40df 5 2024-03-29T11:12:03.574 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:03.595+0000 7f1e7ea43040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:12:03.577 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:03.600+0000 7f84d7221040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:12:03.630 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:03.651+0000 7f1e7ea43040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:12:03.674 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.696+0000 7f9778dc1640 10 monclient: get_monmap_and_config 2024-03-29T11:12:03.674 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.696+0000 7f9778dc1640 10 monclient: build_initial_monmap 2024-03-29T11:12:03.675 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.696+0000 7f9778dc1640 10 monclient: monmap: 2024-03-29T11:12:03.675 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2024-03-29T11:12:03.675 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.675 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:12:03.697795+0000 2024-03-29T11:12:03.675 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:12:03.697795+0000 2024-03-29T11:12:03.675 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:03.675 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.675 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:03.675 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:12:03.675 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:12:03.675 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.685 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.697+0000 7f9778dc1640 1 Processor -- start 2024-03-29T11:12:03.685 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.697+0000 7f9778dc1640 1 -- start start 2024-03-29T11:12:03.685 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.697+0000 7f9778dc1640 10 monclient: init 2024-03-29T11:12:03.685 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.697+0000 7f9778dc1640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:03.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.697+0000 7f9778dc1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:03.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.697+0000 7f9778dc1640 10 monclient(hunting): picked mon.noname-a con 0x7f977409ac40 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:03.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.697+0000 7f9778dc1640 10 monclient(hunting): picked mon.noname-c con 0x7f977409c870 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:03.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.697+0000 7f9778dc1640 10 monclient(hunting): picked mon.noname-b con 0x7f97740b4360 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:03.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.697+0000 7f9778dc1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977409dec0 con 0x7f977409ac40 2024-03-29T11:12:03.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.697+0000 7f9778dc1640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977409da30 con 0x7f977409c870 2024-03-29T11:12:03.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.697+0000 7f9778dc1640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97740b39c0 con 0x7f97740b4360 2024-03-29T11:12:03.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.697+0000 7f9778dc1640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:03.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.697+0000 7f9778dc1640 10 monclient(hunting): authenticate will time out at 913.881940s 2024-03-29T11:12:03.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f9771d9b640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f977409c870 0x7f977413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:59540/0 (socket says 172.21.15.184:59540) 2024-03-29T11:12:03.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f9771d9b640 1 -- 172.21.15.184:0/626564826 learned_addr learned my addr 172.21.15.184:0/626564826 (peer_addr_for_me v1:172.21.15.184:0/0) 2024-03-29T11:12:03.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 1 -- 172.21.15.184:0/626564826 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263332515 0 0) 0x7f977409da30 con 0x7f977409c870 2024-03-29T11:12:03.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:12:03.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 1 -- 172.21.15.184:0/626564826 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f975c003520 con 0x7f977409c870 2024-03-29T11:12:03.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 1 -- 172.21.15.184:0/626564826 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197001129 0 0) 0x7f97740b39c0 con 0x7f97740b4360 2024-03-29T11:12:03.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:12:03.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 1 -- 172.21.15.184:0/626564826 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f977409da30 con 0x7f97740b4360 2024-03-29T11:12:03.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 1 -- 172.21.15.184:0/626564826 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706214944 0 0) 0x7f9764001040 con 0x7f977409ac40 2024-03-29T11:12:03.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:12:03.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 1 -- 172.21.15.184:0/626564826 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97740b39c0 con 0x7f977409ac40 2024-03-29T11:12:03.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 1 -- 172.21.15.184:0/626564826 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 700906847 0 0) 0x7f975c003520 con 0x7f977409c870 2024-03-29T11:12:03.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 1 -- 172.21.15.184:0/626564826 >> v1:172.21.15.170:6790/0 conn(0x7f97740b4360 legacy=0x7f977414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 1 -- 172.21.15.184:0/626564826 >> v1:172.21.15.170:6789/0 conn(0x7f977409ac40 legacy=0x7f97740b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient: _finish_hunting 0 2024-03-29T11:12:03.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 1 monclient: found mon.noname-c 2024-03-29T11:12:03.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:03.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 1 -- 172.21.15.184:0/626564826 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97740b5160 con 0x7f977409c870 2024-03-29T11:12:03.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient: _finish_auth 0 2024-03-29T11:12:03.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient: _check_auth_tickets 2024-03-29T11:12:03.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:03.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f9778dc1640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:12:03.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f9778dc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:03.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 1 -- 172.21.15.184:0/626564826 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9768001690 con 0x7f977409c870 2024-03-29T11:12:03.689 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.689 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:03.689 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient: dump: 2024-03-29T11:12:03.689 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.689 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.689 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.689 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.689 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.689 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.689 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.689 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.689 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.689 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.689 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f9778dc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:03.689 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 1 -- 172.21.15.184:0/626564826 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9768001870 con 0x7f977409c870 2024-03-29T11:12:03.690 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:03.690 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 1 -- 172.21.15.184:0/626564826 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9768001c80 con 0x7f977409c870 2024-03-29T11:12:03.690 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f9778dc1640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:03.690 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f9778dc1640 4 set_mon_vals no callback set 2024-03-29T11:12:03.690 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.690 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:03.690 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.698+0000 7f977159a640 10 monclient: dump: 2024-03-29T11:12:03.690 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.690 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.690 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.690 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.690 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.690 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.690 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.690 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.691 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.691 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.691 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.699+0000 7f9778dc1640 10 monclient: shutdown 2024-03-29T11:12:03.691 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.699+0000 7f9778dc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:03.691 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.699+0000 7f9778dc1640 1 -- 172.21.15.184:0/626564826 >> v1:172.21.15.184:6789/0 conn(0x7f977409c870 legacy=0x7f977413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.691 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.699+0000 7f9778dc1640 1 -- 172.21.15.184:0/626564826 shutdown_connections 2024-03-29T11:12:03.691 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.699+0000 7f9778dc1640 1 -- 172.21.15.184:0/626564826 >> 172.21.15.184:0/626564826 conn(0x7f9774066b00 msgr2=0x7f9774066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:03.691 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.699+0000 7f9778dc1640 1 -- 172.21.15.184:0/626564826 shutdown_connections 2024-03-29T11:12:03.691 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.699+0000 7f9778dc1640 1 -- 172.21.15.184:0/626564826 wait complete. 2024-03-29T11:12:03.691 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.700+0000 7f9778dc1640 10 monclient: build_initial_monmap 2024-03-29T11:12:03.691 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.700+0000 7f9778dc1640 10 monclient: monmap: 2024-03-29T11:12:03.691 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2024-03-29T11:12:03.691 INFO:teuthology.orchestra.run.smithi184.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:03.691 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 0.000000 2024-03-29T11:12:03.691 INFO:teuthology.orchestra.run.smithi184.stderr:created 0.000000 2024-03-29T11:12:03.692 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:03.692 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.692 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:03.692 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:12:03.692 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:12:03.692 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.692 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.700+0000 7f9778dc1640 1 librados: starting msgr at 2024-03-29T11:12:03.692 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.700+0000 7f9778dc1640 1 librados: starting objecter 2024-03-29T11:12:03.692 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.700+0000 7f9778dc1640 1 Processor -- start 2024-03-29T11:12:03.692 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.700+0000 7f9778dc1640 1 -- start start 2024-03-29T11:12:03.692 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.700+0000 7f9778dc1640 1 librados: setting wanted keys 2024-03-29T11:12:03.692 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.700+0000 7f9778dc1640 1 librados: calling monclient init 2024-03-29T11:12:03.692 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.700+0000 7f9778dc1640 10 monclient: init 2024-03-29T11:12:03.692 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9778dc1640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:03.692 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9778dc1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:03.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9778dc1640 10 monclient(hunting): picked mon.noname-a con 0x7f977409ac40 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:03.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9778dc1640 10 monclient(hunting): picked mon.noname-b con 0x7f97741d77f0 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:03.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9778dc1640 10 monclient(hunting): picked mon.noname-c con 0x7f9774142b60 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:03.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9778dc1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9774140380 con 0x7f977409ac40 2024-03-29T11:12:03.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9778dc1640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97741408a0 con 0x7f97741d77f0 2024-03-29T11:12:03.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9778dc1640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977413ffa0 con 0x7f9774142b60 2024-03-29T11:12:03.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9778dc1640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:03.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9778dc1640 10 monclient(hunting): authenticate will time out at 913.885224s 2024-03-29T11:12:03.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9771d9b640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f97741d77f0 0x7f97740b0f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:59554/0 (socket says 172.21.15.184:59554) 2024-03-29T11:12:03.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9771d9b640 1 -- 172.21.15.184:0/3012781574 learned_addr learned my addr 172.21.15.184:0/3012781574 (peer_addr_for_me v1:172.21.15.184:0/0) 2024-03-29T11:12:03.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 client.?.objecter ms_handle_connect 0x7f97741d77f0 2024-03-29T11:12:03.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:03.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 client.?.objecter ms_handle_connect 0x7f977409ac40 2024-03-29T11:12:03.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:03.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 client.?.objecter ms_handle_connect 0x7f9774142b60 2024-03-29T11:12:03.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:03.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754083 0 0) 0x7f97741408a0 con 0x7f97741d77f0 2024-03-29T11:12:03.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:12:03.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9754004eb0 con 0x7f97741d77f0 2024-03-29T11:12:03.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462148980 0 0) 0x7f9774140380 con 0x7f977409ac40 2024-03-29T11:12:03.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:12:03.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97741408a0 con 0x7f977409ac40 2024-03-29T11:12:03.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906547629 0 0) 0x7f977413ffa0 con 0x7f9774142b60 2024-03-29T11:12:03.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:12:03.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9774140380 con 0x7f9774142b60 2024-03-29T11:12:03.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1667617 0 0) 0x7f9774140380 con 0x7f9774142b60 2024-03-29T11:12:03.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f977413ffa0 con 0x7f9774142b60 2024-03-29T11:12:03.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3892742425 0 0) 0x7f97741408a0 con 0x7f977409ac40 2024-03-29T11:12:03.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9774140380 con 0x7f977409ac40 2024-03-29T11:12:03.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 299927091 0 0) 0x7f9754004eb0 con 0x7f97741d77f0 2024-03-29T11:12:03.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97741408a0 con 0x7f97741d77f0 2024-03-29T11:12:03.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f976c001a50 con 0x7f9774142b60 2024-03-29T11:12:03.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:03.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): dump: 2024-03-29T11:12:03.696 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.696 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.696 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.696 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.696 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.696 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.696 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.696 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.696 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.696 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:03.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9764001930 con 0x7f977409ac40 2024-03-29T11:12:03.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:03.697 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): dump: 2024-03-29T11:12:03.697 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.697 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.697 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.697 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.697 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.697 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.697 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.697 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.697 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.697 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.697 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9768001b80 con 0x7f97741d77f0 2024-03-29T11:12:03.697 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.698 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:03.698 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient(hunting): dump: 2024-03-29T11:12:03.698 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.698 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.698 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.698 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.698 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.698 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.698 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.698 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.698 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.698 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.698 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 954426165 0 0) 0x7f977413ffa0 con 0x7f9774142b60 2024-03-29T11:12:03.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 >> v1:172.21.15.184:6789/0 conn(0x7f97741d77f0 legacy=0x7f97740b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 >> v1:172.21.15.170:6789/0 conn(0x7f977409ac40 legacy=0x7f97741d70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:03.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 monclient: found mon.c 2024-03-29T11:12:03.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:03.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:03.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9774140130 con 0x7f9774142b60 2024-03-29T11:12:03.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient: _finish_auth 0 2024-03-29T11:12:03.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:03.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9762ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:03.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.701+0000 7f9778dc1640 5 monclient: authenticate success, global_id 4148 2024-03-29T11:12:03.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9778dc1640 10 monclient: _renew_subs 2024-03-29T11:12:03.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9778dc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:03.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9778dc1640 1 -- 172.21.15.184:0/3012781574 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97741462a0 con 0x7f9774142b60 2024-03-29T11:12:03.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9778dc1640 10 client.4148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:03.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9778dc1640 10 monclient: _renew_subs 2024-03-29T11:12:03.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9778dc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:03.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9778dc1640 1 -- 172.21.15.184:0/3012781574 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9774141290 con 0x7f9774142b60 2024-03-29T11:12:03.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9778dc1640 1 librados: init done 2024-03-29T11:12:03.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f976c001bb0 con 0x7f9774142b60 2024-03-29T11:12:03.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9762ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:03.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f976c003970 con 0x7f9774142b60 2024-03-29T11:12:03.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9762ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9762ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:03.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9770d99640 4 set_mon_vals no callback set 2024-03-29T11:12:03.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9762ffd640 10 monclient: dump: 2024-03-29T11:12:03.700 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.700 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.700 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.701 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.701 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.701 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.701 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.701 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.701 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.701 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.701 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f976c003ee0 con 0x7f9774142b60 2024-03-29T11:12:03.701 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 4140717990 0 0) 0x7f976c004710 con 0x7f9774142b60 2024-03-29T11:12:03.701 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9762ffd640 10 client.4148.objecter ms_dispatch 0x7f9774000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:12:03.701 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9762ffd640 3 client.4148.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:12:03.701 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9762ffd640 3 client.4148.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:12:03.701 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9762ffd640 20 client.4148.objecter dump_active .. 0 homeless 2024-03-29T11:12:03.702 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f1e7ea43040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:12:03.702 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9778dc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:03.702 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9778dc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:03.702 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9778dc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:03.703 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.702+0000 7f9778dc1640 1 -- 172.21.15.184:0/3012781574 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9758005080 con 0x7f9774142b60 2024-03-29T11:12:03.703 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.705+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f976c004b00 con 0x7f9774142b60 2024-03-29T11:12:03.703 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.705+0000 7f9762ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:03.703 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.705+0000 7f9762ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:03.755 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.777+0000 7f9778dc1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9442ffc6-a927-47de-a9f3-7f8cb0ed40df", "id": 5}] 2024-03-29T11:12:03.755 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.777+0000 7f9778dc1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9442ffc6-a927-47de-a9f3-7f8cb0ed40df", "id": 5}] 2024-03-29T11:12:03.756 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.777+0000 7f9778dc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:03.756 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.777+0000 7f9778dc1640 1 -- 172.21.15.184:0/3012781574 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd new", "uuid": "9442ffc6-a927-47de-a9f3-7f8cb0ed40df", "id": 5} v 0) v1 -- 0x7f97580058b0 con 0x7f9774142b60 2024-03-29T11:12:03.758 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.780+0000 7f9762ffd640 1 -- 172.21.15.184:0/3012781574 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9442ffc6-a927-47de-a9f3-7f8cb0ed40df", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1602147136 0 1988412176) 0x7f976c004190 con 0x7f9774142b60 2024-03-29T11:12:03.758 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.780+0000 7f9762ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9442ffc6-a927-47de-a9f3-7f8cb0ed40df", "id": 5}] 2024-03-29T11:12:03.758 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.780+0000 7f9762ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:03.759 INFO:teuthology.orchestra.run.smithi184.stdout:5 2024-03-29T11:12:03.760 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.781+0000 7f9778dc1640 10 librados: watch_flush enter 2024-03-29T11:12:03.760 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.781+0000 7f9778dc1640 10 librados: watch_flush exit 2024-03-29T11:12:03.760 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.781+0000 7f9778dc1640 20 client.4148.objecter shutdown clearing up homeless session... 2024-03-29T11:12:03.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.781+0000 7f9778dc1640 10 client.4148.objecter successfully canceled tick 2024-03-29T11:12:03.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.781+0000 7f9778dc1640 10 monclient: shutdown 2024-03-29T11:12:03.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.781+0000 7f9778dc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:03.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.781+0000 7f9778dc1640 1 -- 172.21.15.184:0/3012781574 >> v1:172.21.15.170:6790/0 conn(0x7f9774142b60 legacy=0x7f97741e2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.781+0000 7f9778dc1640 1 -- 172.21.15.184:0/3012781574 shutdown_connections 2024-03-29T11:12:03.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.781+0000 7f9778dc1640 1 -- 172.21.15.184:0/3012781574 >> 172.21.15.184:0/3012781574 conn(0x7f9774066b00 msgr2=0x7f9774066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:03.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.781+0000 7f9778dc1640 1 -- 172.21.15.184:0/3012781574 shutdown_connections 2024-03-29T11:12:03.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.781+0000 7f9778dc1640 1 -- 172.21.15.184:0/3012781574 wait complete. 2024-03-29T11:12:03.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.781+0000 7f9778dc1640 1 librados: shutdown 2024-03-29T11:12:03.770 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 29872cd7-f7bf-4bc0-b70f-e0ff589d69f3 6 2024-03-29T11:12:03.870 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.892+0000 7f9c28a51640 10 monclient: get_monmap_and_config 2024-03-29T11:12:03.870 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.892+0000 7f9c28a51640 10 monclient: build_initial_monmap 2024-03-29T11:12:03.870 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.892+0000 7f9c28a51640 10 monclient: monmap: 2024-03-29T11:12:03.870 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2024-03-29T11:12:03.871 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.871 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:12:03.893395+0000 2024-03-29T11:12:03.871 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:12:03.893395+0000 2024-03-29T11:12:03.871 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:03.871 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.871 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:03.871 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:12:03.871 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:12:03.871 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.880 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.892+0000 7f9c28a51640 1 Processor -- start 2024-03-29T11:12:03.880 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.892+0000 7f9c28a51640 1 -- start start 2024-03-29T11:12:03.880 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.892+0000 7f9c28a51640 10 monclient: init 2024-03-29T11:12:03.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c28a51640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:03.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c28a51640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:03.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c28a51640 10 monclient(hunting): picked mon.noname-a con 0x7f9c2409ac40 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:03.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c28a51640 10 monclient(hunting): picked mon.noname-c con 0x7f9c2409c870 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:03.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c28a51640 10 monclient(hunting): picked mon.noname-b con 0x7f9c240b4360 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:03.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c28a51640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c2409dec0 con 0x7f9c2409ac40 2024-03-29T11:12:03.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c28a51640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c2409da30 con 0x7f9c2409c870 2024-03-29T11:12:03.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c28a51640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c240b39c0 con 0x7f9c240b4360 2024-03-29T11:12:03.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c28a51640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:03.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c28a51640 10 monclient(hunting): authenticate will time out at 914.077581s 2024-03-29T11:12:03.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c1bfff640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f9c2409c870 0x7f9c2413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:59562/0 (socket says 172.21.15.184:59562) 2024-03-29T11:12:03.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c1bfff640 1 -- 172.21.15.184:0/3176634938 learned_addr learned my addr 172.21.15.184:0/3176634938 (peer_addr_for_me v1:172.21.15.184:0/0) 2024-03-29T11:12:03.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c21d9b640 1 -- 172.21.15.184:0/3176634938 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474845423 0 0) 0x7f9c2409da30 con 0x7f9c2409c870 2024-03-29T11:12:03.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c21d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c21d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c21d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c21d9b640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:12:03.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c21d9b640 1 -- 172.21.15.184:0/3176634938 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c10003520 con 0x7f9c2409c870 2024-03-29T11:12:03.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c21d9b640 1 -- 172.21.15.184:0/3176634938 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598960124 0 0) 0x7f9c14001110 con 0x7f9c2409ac40 2024-03-29T11:12:03.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c21d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c21d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c21d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c21d9b640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:12:03.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.893+0000 7f9c21d9b640 1 -- 172.21.15.184:0/3176634938 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c2409da30 con 0x7f9c2409ac40 2024-03-29T11:12:03.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 1 -- 172.21.15.184:0/3176634938 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667356558 0 0) 0x7f9c240b39c0 con 0x7f9c240b4360 2024-03-29T11:12:03.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:12:03.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 1 -- 172.21.15.184:0/3176634938 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c14001110 con 0x7f9c240b4360 2024-03-29T11:12:03.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 1 -- 172.21.15.184:0/3176634938 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2561988584 0 0) 0x7f9c10003520 con 0x7f9c2409c870 2024-03-29T11:12:03.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 1 -- 172.21.15.184:0/3176634938 >> v1:172.21.15.170:6790/0 conn(0x7f9c240b4360 legacy=0x7f9c2414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 1 -- 172.21.15.184:0/3176634938 >> v1:172.21.15.170:6789/0 conn(0x7f9c2409ac40 legacy=0x7f9c240b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:12:03.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 1 monclient: found mon.noname-c 2024-03-29T11:12:03.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:03.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 1 -- 172.21.15.184:0/3176634938 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c240b5160 con 0x7f9c2409c870 2024-03-29T11:12:03.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 10 monclient: _finish_auth 0 2024-03-29T11:12:03.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:12:03.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:03.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 1 -- 172.21.15.184:0/3176634938 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9c0c001690 con 0x7f9c2409c870 2024-03-29T11:12:03.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c28a51640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:12:03.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c28a51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:03.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.884 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:03.884 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 10 monclient: dump: 2024-03-29T11:12:03.884 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.884 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.884 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.884 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.884 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.884 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.884 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.884 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.884 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.884 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.884 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c28a51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:03.884 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 1 -- 172.21.15.184:0/3176634938 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c0c001870 con 0x7f9c2409c870 2024-03-29T11:12:03.884 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:03.884 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 1 -- 172.21.15.184:0/3176634938 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9c0c001c80 con 0x7f9c2409c870 2024-03-29T11:12:03.885 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c28a51640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:03.885 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c28a51640 4 set_mon_vals no callback set 2024-03-29T11:12:03.885 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.885 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:03.885 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c21d9b640 10 monclient: dump: 2024-03-29T11:12:03.885 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.885 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.885 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.885 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.885 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.885 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.885 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.885 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.885 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.885 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.885 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c28a51640 10 monclient: shutdown 2024-03-29T11:12:03.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c28a51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:03.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c28a51640 1 -- 172.21.15.184:0/3176634938 >> v1:172.21.15.184:6789/0 conn(0x7f9c2409c870 legacy=0x7f9c2413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c28a51640 1 -- 172.21.15.184:0/3176634938 shutdown_connections 2024-03-29T11:12:03.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c28a51640 1 -- 172.21.15.184:0/3176634938 >> 172.21.15.184:0/3176634938 conn(0x7f9c24066b00 msgr2=0x7f9c24066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:03.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c28a51640 1 -- 172.21.15.184:0/3176634938 shutdown_connections 2024-03-29T11:12:03.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.894+0000 7f9c28a51640 1 -- 172.21.15.184:0/3176634938 wait complete. 2024-03-29T11:12:03.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.895+0000 7f9c28a51640 10 monclient: build_initial_monmap 2024-03-29T11:12:03.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.895+0000 7f9c28a51640 10 monclient: monmap: 2024-03-29T11:12:03.886 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2024-03-29T11:12:03.886 INFO:teuthology.orchestra.run.smithi184.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:03.886 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 0.000000 2024-03-29T11:12:03.886 INFO:teuthology.orchestra.run.smithi184.stderr:created 0.000000 2024-03-29T11:12:03.886 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:03.887 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.887 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:03.887 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:12:03.887 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:12:03.887 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.887 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.895+0000 7f9c28a51640 1 librados: starting msgr at 2024-03-29T11:12:03.887 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.895+0000 7f9c28a51640 1 librados: starting objecter 2024-03-29T11:12:03.887 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.895+0000 7f9c28a51640 1 Processor -- start 2024-03-29T11:12:03.887 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c28a51640 1 -- start start 2024-03-29T11:12:03.887 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c28a51640 1 librados: setting wanted keys 2024-03-29T11:12:03.887 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c28a51640 1 librados: calling monclient init 2024-03-29T11:12:03.887 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c28a51640 10 monclient: init 2024-03-29T11:12:03.887 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c28a51640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:03.887 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c28a51640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:12:03.887 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c28a51640 10 monclient(hunting): picked mon.noname-c con 0x7f9c2409ac40 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:03.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c28a51640 10 monclient(hunting): picked mon.noname-a con 0x7f9c240acf70 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:03.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c28a51640 10 monclient(hunting): picked mon.noname-b con 0x7f9c241e0100 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:03.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c28a51640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c241d7800 con 0x7f9c240acf70 2024-03-29T11:12:03.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c28a51640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c240ade40 con 0x7f9c241e0100 2024-03-29T11:12:03.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c28a51640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c240ae120 con 0x7f9c2409ac40 2024-03-29T11:12:03.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c28a51640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:03.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c28a51640 10 monclient(hunting): authenticate will time out at 914.080566s 2024-03-29T11:12:03.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c22d9d640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f9c241e0100 0x7f9c241e24f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:59566/0 (socket says 172.21.15.184:59566) 2024-03-29T11:12:03.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c22d9d640 1 -- 172.21.15.184:0/2769947823 learned_addr learned my addr 172.21.15.184:0/2769947823 (peer_addr_for_me v1:172.21.15.184:0/0) 2024-03-29T11:12:03.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 10 client.?.objecter ms_handle_connect 0x7f9c241e0100 2024-03-29T11:12:03.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:03.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 10 client.?.objecter ms_handle_connect 0x7f9c240acf70 2024-03-29T11:12:03.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:03.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 10 client.?.objecter ms_handle_connect 0x7f9c2409ac40 2024-03-29T11:12:03.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:03.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964469377 0 0) 0x7f9c240ade40 con 0x7f9c241e0100 2024-03-29T11:12:03.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:12:03.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c04004eb0 con 0x7f9c241e0100 2024-03-29T11:12:03.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107992345 0 0) 0x7f9c241d7800 con 0x7f9c240acf70 2024-03-29T11:12:03.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:12:03.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c240ade40 con 0x7f9c240acf70 2024-03-29T11:12:03.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044477187 0 0) 0x7f9c240ae120 con 0x7f9c2409ac40 2024-03-29T11:12:03.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:12:03.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.896+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c241d7800 con 0x7f9c2409ac40 2024-03-29T11:12:03.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2383251933 0 0) 0x7f9c04004eb0 con 0x7f9c241e0100 2024-03-29T11:12:03.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c240ae120 con 0x7f9c241e0100 2024-03-29T11:12:03.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9c1c001c10 con 0x7f9c241e0100 2024-03-29T11:12:03.892 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.892 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:03.892 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 10 monclient(hunting): dump: 2024-03-29T11:12:03.892 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.892 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.892 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.892 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.892 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.892 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.892 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.892 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.892 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.892 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.892 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:03.892 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4259394899 0 0) 0x7f9c240ade40 con 0x7f9c240acf70 2024-03-29T11:12:03.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c04004eb0 con 0x7f9c240acf70 2024-03-29T11:12:03.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9c0c001b80 con 0x7f9c240acf70 2024-03-29T11:12:03.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:03.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 10 monclient(hunting): dump: 2024-03-29T11:12:03.893 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.893 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.893 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.893 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.893 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.893 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.893 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.893 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.893 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.893 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3879138898 0 0) 0x7f9c241d7800 con 0x7f9c2409ac40 2024-03-29T11:12:03.894 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c240ade40 con 0x7f9c2409ac40 2024-03-29T11:12:03.894 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9c14001860 con 0x7f9c2409ac40 2024-03-29T11:12:03.894 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.894 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:03.894 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 10 monclient(hunting): dump: 2024-03-29T11:12:03.894 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.894 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.894 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.894 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.894 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.894 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.894 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.894 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.895 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.895 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 968737987 0 0) 0x7f9c240ae120 con 0x7f9c241e0100 2024-03-29T11:12:03.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 >> v1:172.21.15.170:6790/0 conn(0x7f9c2409ac40 legacy=0x7f9c24142760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 >> v1:172.21.15.170:6789/0 conn(0x7f9c240acf70 legacy=0x7f9c241d7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:03.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 1 monclient: found mon.b 2024-03-29T11:12:03.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:03.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2413ffa0 con 0x7f9c241e0100 2024-03-29T11:12:03.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 10 monclient: _finish_auth 0 2024-03-29T11:12:03.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:03.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:03.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c28a51640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:12:03.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c28a51640 10 monclient: _renew_subs 2024-03-29T11:12:03.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c28a51640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c28a51640 1 -- 172.21.15.184:0/2769947823 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c241d83a0 con 0x7f9c241e0100 2024-03-29T11:12:03.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c28a51640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:03.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c28a51640 10 monclient: _renew_subs 2024-03-29T11:12:03.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c28a51640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c28a51640 1 -- 172.21.15.184:0/2769947823 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c241e3df0 con 0x7f9c241e0100 2024-03-29T11:12:03.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c28a51640 1 librados: init done 2024-03-29T11:12:03.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c1c003990 con 0x7f9c241e0100 2024-03-29T11:12:03.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:03.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9c1c003d60 con 0x7f9c241e0100 2024-03-29T11:12:03.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:03.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c2159a640 4 set_mon_vals no callback set 2024-03-29T11:12:03.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 10 monclient: dump: 2024-03-29T11:12:03.896 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:03.897 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:03.897 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.897 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:03.897 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.897 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:03.897 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:03.897 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:03.897 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:03.897 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:03.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9c1c0042d0 con 0x7f9c241e0100 2024-03-29T11:12:03.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1601235926 0 0) 0x7f9c1c006b20 con 0x7f9c241e0100 2024-03-29T11:12:03.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 10 client.4141.objecter ms_dispatch 0x7f9c24000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:12:03.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 3 client.4141.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:12:03.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 3 client.4141.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:12:03.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.897+0000 7f9c1affd640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T11:12:03.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.898+0000 7f9c28a51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:03.898 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.898+0000 7f9c28a51640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:03.898 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.898+0000 7f9c28a51640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.898 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.898+0000 7f9c28a51640 1 -- 172.21.15.184:0/2769947823 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c08005080 con 0x7f9c241e0100 2024-03-29T11:12:03.898 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.900+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9c1c006530 con 0x7f9c241e0100 2024-03-29T11:12:03.898 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.900+0000 7f9c1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:03.898 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.900+0000 7f9c1affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:03.949 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.970+0000 7f9c28a51640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "29872cd7-f7bf-4bc0-b70f-e0ff589d69f3", "id": 6}] 2024-03-29T11:12:03.949 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.970+0000 7f9c28a51640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "29872cd7-f7bf-4bc0-b70f-e0ff589d69f3", "id": 6}] 2024-03-29T11:12:03.949 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.970+0000 7f9c28a51640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:03.949 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.970+0000 7f9c28a51640 1 -- 172.21.15.184:0/2769947823 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd new", "uuid": "29872cd7-f7bf-4bc0-b70f-e0ff589d69f3", "id": 6} v 0) v1 -- 0x7f9c080058b0 con 0x7f9c241e0100 2024-03-29T11:12:03.952 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.974+0000 7f9c1affd640 1 -- 172.21.15.184:0/2769947823 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "29872cd7-f7bf-4bc0-b70f-e0ff589d69f3", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1313665245 0 1113811849) 0x7f9c1c006790 con 0x7f9c241e0100 2024-03-29T11:12:03.952 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.974+0000 7f9c1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "29872cd7-f7bf-4bc0-b70f-e0ff589d69f3", "id": 6}] 2024-03-29T11:12:03.952 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.974+0000 7f9c1affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:03.953 INFO:teuthology.orchestra.run.smithi184.stdout:6 2024-03-29T11:12:03.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.975+0000 7f9c28a51640 10 librados: watch_flush enter 2024-03-29T11:12:03.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.975+0000 7f9c28a51640 10 librados: watch_flush exit 2024-03-29T11:12:03.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.975+0000 7f9c28a51640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T11:12:03.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.975+0000 7f9c28a51640 10 client.4141.objecter successfully canceled tick 2024-03-29T11:12:03.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.975+0000 7f9c28a51640 10 monclient: shutdown 2024-03-29T11:12:03.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.975+0000 7f9c28a51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:03.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.975+0000 7f9c28a51640 1 -- 172.21.15.184:0/2769947823 >> v1:172.21.15.184:6789/0 conn(0x7f9c241e0100 legacy=0x7f9c241e24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.975+0000 7f9c28a51640 1 -- 172.21.15.184:0/2769947823 shutdown_connections 2024-03-29T11:12:03.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.975+0000 7f9c28a51640 1 -- 172.21.15.184:0/2769947823 >> 172.21.15.184:0/2769947823 conn(0x7f9c24066b00 msgr2=0x7f9c2413e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:03.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.975+0000 7f9c28a51640 1 -- 172.21.15.184:0/2769947823 shutdown_connections 2024-03-29T11:12:03.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.975+0000 7f9c28a51640 1 -- 172.21.15.184:0/2769947823 wait complete. 2024-03-29T11:12:03.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:03.975+0000 7f9c28a51640 1 librados: shutdown 2024-03-29T11:12:03.958 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:03.981+0000 7f84d7221040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:12:03.964 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 71f6869e-a468-4f2b-8f4b-c9c6005266b4 7 2024-03-29T11:12:04.063 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.085+0000 7f691a1cb640 10 monclient: get_monmap_and_config 2024-03-29T11:12:04.063 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.085+0000 7f691a1cb640 10 monclient: build_initial_monmap 2024-03-29T11:12:04.063 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.085+0000 7f691a1cb640 10 monclient: monmap: 2024-03-29T11:12:04.064 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2024-03-29T11:12:04.064 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:04.064 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:12:04.086650+0000 2024-03-29T11:12:04.064 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:12:04.086650+0000 2024-03-29T11:12:04.064 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:04.064 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:04.064 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:04.064 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:12:04.064 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:12:04.064 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:04.075 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.086+0000 7f691a1cb640 1 Processor -- start 2024-03-29T11:12:04.076 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.086+0000 7f691a1cb640 1 -- start start 2024-03-29T11:12:04.076 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.086+0000 7f691a1cb640 10 monclient: init 2024-03-29T11:12:04.076 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.086+0000 7f691a1cb640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:04.076 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.086+0000 7f691a1cb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:04.076 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.086+0000 7f691a1cb640 10 monclient(hunting): picked mon.noname-c con 0x7f691409ac40 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:04.076 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.086+0000 7f691a1cb640 10 monclient(hunting): picked mon.noname-a con 0x7f691409c870 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:04.076 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.086+0000 7f691a1cb640 10 monclient(hunting): picked mon.noname-b con 0x7f69140b4360 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:04.076 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.086+0000 7f691a1cb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691409dec0 con 0x7f691409c870 2024-03-29T11:12:04.076 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.086+0000 7f691a1cb640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691409da30 con 0x7f691409ac40 2024-03-29T11:12:04.076 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.086+0000 7f691a1cb640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69140b39c0 con 0x7f69140b4360 2024-03-29T11:12:04.076 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.086+0000 7f691a1cb640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:04.076 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.086+0000 7f691a1cb640 10 monclient(hunting): authenticate will time out at 914.270707s 2024-03-29T11:12:04.076 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.086+0000 7f69137fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f691409ac40 0x7f69140b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:59568/0 (socket says 172.21.15.184:59568) 2024-03-29T11:12:04.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.086+0000 7f69137fe640 1 -- 172.21.15.184:0/4226397872 learned_addr learned my addr 172.21.15.184:0/4226397872 (peer_addr_for_me v1:172.21.15.184:0/0) 2024-03-29T11:12:04.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 1 -- 172.21.15.184:0/4226397872 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381067705 0 0) 0x7f691409da30 con 0x7f691409ac40 2024-03-29T11:12:04.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:04.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:04.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:04.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:12:04.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 1 -- 172.21.15.184:0/4226397872 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68fc003520 con 0x7f691409ac40 2024-03-29T11:12:04.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 1 -- 172.21.15.184:0/4226397872 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345391674 0 0) 0x7f69140b39c0 con 0x7f69140b4360 2024-03-29T11:12:04.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:04.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:04.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:04.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient(hunting): my global_id is 4157 2024-03-29T11:12:04.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 1 -- 172.21.15.184:0/4226397872 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f691409da30 con 0x7f69140b4360 2024-03-29T11:12:04.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 1 -- 172.21.15.184:0/4226397872 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742993367 0 0) 0x7f6908001110 con 0x7f691409c870 2024-03-29T11:12:04.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:04.078 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:04.078 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:04.078 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:12:04.078 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 1 -- 172.21.15.184:0/4226397872 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69140b39c0 con 0x7f691409c870 2024-03-29T11:12:04.078 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 1 -- 172.21.15.184:0/4226397872 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3099682300 0 0) 0x7f68fc003520 con 0x7f691409ac40 2024-03-29T11:12:04.078 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 1 -- 172.21.15.184:0/4226397872 >> v1:172.21.15.170:6790/0 conn(0x7f69140b4360 legacy=0x7f691414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.078 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 1 -- 172.21.15.184:0/4226397872 >> v1:172.21.15.170:6789/0 conn(0x7f691409c870 legacy=0x7f691413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.078 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient: _finish_hunting 0 2024-03-29T11:12:04.078 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 1 monclient: found mon.noname-c 2024-03-29T11:12:04.078 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:04.078 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:12:04.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 1 -- 172.21.15.184:0/4226397872 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69140b5160 con 0x7f691409ac40 2024-03-29T11:12:04.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient: _finish_auth 0 2024-03-29T11:12:04.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient: _check_auth_tickets 2024-03-29T11:12:04.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:04.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 1 -- 172.21.15.184:0/4226397872 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f6904001690 con 0x7f691409ac40 2024-03-29T11:12:04.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f691a1cb640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:12:04.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f691a1cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:04.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:04.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:04.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient: dump: 2024-03-29T11:12:04.079 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:04.079 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:04.079 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.079 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.079 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:04.080 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:04.080 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:04.080 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:04.080 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:04.080 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:04.080 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f691a1cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:04.080 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 1 -- 172.21.15.184:0/4226397872 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6904001870 con 0x7f691409ac40 2024-03-29T11:12:04.080 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:04.080 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 1 -- 172.21.15.184:0/4226397872 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f6904001c80 con 0x7f691409ac40 2024-03-29T11:12:04.080 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f691a1cb640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:04.080 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f691a1cb640 4 set_mon_vals no callback set 2024-03-29T11:12:04.080 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:04.080 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:04.080 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.087+0000 7f69127fc640 10 monclient: dump: 2024-03-29T11:12:04.080 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:04.080 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:04.081 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.081 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.081 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:04.081 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:04.081 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:04.081 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:04.081 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:04.081 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:04.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.088+0000 7f691a1cb640 10 monclient: shutdown 2024-03-29T11:12:04.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.088+0000 7f691a1cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:04.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.088+0000 7f691a1cb640 1 -- 172.21.15.184:0/4226397872 >> v1:172.21.15.184:6789/0 conn(0x7f691409ac40 legacy=0x7f69140b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.088+0000 7f691a1cb640 1 -- 172.21.15.184:0/4226397872 shutdown_connections 2024-03-29T11:12:04.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.088+0000 7f691a1cb640 1 -- 172.21.15.184:0/4226397872 >> 172.21.15.184:0/4226397872 conn(0x7f6914066b00 msgr2=0x7f6914066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:04.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.088+0000 7f691a1cb640 1 -- 172.21.15.184:0/4226397872 shutdown_connections 2024-03-29T11:12:04.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.088+0000 7f691a1cb640 1 -- 172.21.15.184:0/4226397872 wait complete. 2024-03-29T11:12:04.082 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.089+0000 7f691a1cb640 10 monclient: build_initial_monmap 2024-03-29T11:12:04.082 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.089+0000 7f691a1cb640 10 monclient: monmap: 2024-03-29T11:12:04.082 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2024-03-29T11:12:04.082 INFO:teuthology.orchestra.run.smithi184.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:04.082 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 0.000000 2024-03-29T11:12:04.082 INFO:teuthology.orchestra.run.smithi184.stderr:created 0.000000 2024-03-29T11:12:04.082 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:04.082 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:04.082 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:04.082 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:12:04.082 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:12:04.082 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:04.082 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.089+0000 7f691a1cb640 1 librados: starting msgr at 2024-03-29T11:12:04.082 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.089+0000 7f691a1cb640 1 librados: starting objecter 2024-03-29T11:12:04.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.089+0000 7f691a1cb640 1 Processor -- start 2024-03-29T11:12:04.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.089+0000 7f691a1cb640 1 -- start start 2024-03-29T11:12:04.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.089+0000 7f691a1cb640 1 librados: setting wanted keys 2024-03-29T11:12:04.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.089+0000 7f691a1cb640 1 librados: calling monclient init 2024-03-29T11:12:04.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.089+0000 7f691a1cb640 10 monclient: init 2024-03-29T11:12:04.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f691a1cb640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:04.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f691a1cb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:12:04.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f691a1cb640 10 monclient(hunting): picked mon.noname-b con 0x7f691409ac40 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:04.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f691a1cb640 10 monclient(hunting): picked mon.noname-c con 0x7f6914138f60 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:04.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f691a1cb640 10 monclient(hunting): picked mon.noname-a con 0x7f69140afdc0 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:04.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f691a1cb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69141db610 con 0x7f69140afdc0 2024-03-29T11:12:04.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f691a1cb640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6914139a00 con 0x7f691409ac40 2024-03-29T11:12:04.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f691a1cb640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69140b0230 con 0x7f6914138f60 2024-03-29T11:12:04.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f691a1cb640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:04.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f691a1cb640 10 monclient(hunting): authenticate will time out at 914.274143s 2024-03-29T11:12:04.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f69137fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f691409ac40 0x7f69141dac80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:59578/0 (socket says 172.21.15.184:59578) 2024-03-29T11:12:04.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f69137fe640 1 -- 172.21.15.184:0/3810798893 learned_addr learned my addr 172.21.15.184:0/3810798893 (peer_addr_for_me v1:172.21.15.184:0/0) 2024-03-29T11:12:04.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 client.?.objecter ms_handle_connect 0x7f691409ac40 2024-03-29T11:12:04.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:04.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 client.?.objecter ms_handle_connect 0x7f69140afdc0 2024-03-29T11:12:04.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:04.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 client.?.objecter ms_handle_connect 0x7f6914138f60 2024-03-29T11:12:04.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:04.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670251469 0 0) 0x7f6914139a00 con 0x7f691409ac40 2024-03-29T11:12:04.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:04.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:04.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:04.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:12:04.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6900004eb0 con 0x7f691409ac40 2024-03-29T11:12:04.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661824057 0 0) 0x7f69141db610 con 0x7f69140afdc0 2024-03-29T11:12:04.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:04.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:04.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:04.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): my global_id is 4161 2024-03-29T11:12:04.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6914139a00 con 0x7f69140afdc0 2024-03-29T11:12:04.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794564942 0 0) 0x7f69140b0230 con 0x7f6914138f60 2024-03-29T11:12:04.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:04.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:04.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:04.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): my global_id is 4160 2024-03-29T11:12:04.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69141db610 con 0x7f6914138f60 2024-03-29T11:12:04.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1900898704 0 0) 0x7f6900004eb0 con 0x7f691409ac40 2024-03-29T11:12:04.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69140b0230 con 0x7f691409ac40 2024-03-29T11:12:04.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f6904001bc0 con 0x7f691409ac40 2024-03-29T11:12:04.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:04.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:04.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): dump: 2024-03-29T11:12:04.086 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:04.086 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:04.086 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.086 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.086 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:04.086 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:04.086 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:04.086 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:04.086 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:04.086 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:04.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:04.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 784739843 0 0) 0x7f6914139a00 con 0x7f69140afdc0 2024-03-29T11:12:04.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6900004eb0 con 0x7f69140afdc0 2024-03-29T11:12:04.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3903380163 0 0) 0x7f69141db610 con 0x7f6914138f60 2024-03-29T11:12:04.087 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6914139a00 con 0x7f6914138f60 2024-03-29T11:12:04.087 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f690c001a10 con 0x7f69140afdc0 2024-03-29T11:12:04.087 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:04.087 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:04.087 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): dump: 2024-03-29T11:12:04.087 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:04.087 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:04.087 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.087 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.087 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:04.087 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:04.087 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:04.087 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:04.087 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:04.087 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:04.088 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f6908001860 con 0x7f6914138f60 2024-03-29T11:12:04.088 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:04.088 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:04.088 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient(hunting): dump: 2024-03-29T11:12:04.088 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:04.088 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:04.088 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.088 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.088 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:04.088 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:04.088 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:04.088 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:04.088 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:04.088 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:04.088 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1002975824 0 0) 0x7f69140b0230 con 0x7f691409ac40 2024-03-29T11:12:04.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 >> v1:172.21.15.170:6790/0 conn(0x7f6914138f60 legacy=0x7f69140af6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 >> v1:172.21.15.170:6789/0 conn(0x7f69140afdc0 legacy=0x7f69141e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:12:04.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 monclient: found mon.b 2024-03-29T11:12:04.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:04.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:04.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69140b03c0 con 0x7f691409ac40 2024-03-29T11:12:04.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient: _finish_auth 0 2024-03-29T11:12:04.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:12:04.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f68fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:04.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f691a1cb640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:12:04.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f691a1cb640 10 monclient: _renew_subs 2024-03-29T11:12:04.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f691a1cb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:04.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f691a1cb640 1 -- 172.21.15.184:0/3810798893 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69141e36e0 con 0x7f691409ac40 2024-03-29T11:12:04.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f691a1cb640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:04.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f691a1cb640 10 monclient: _renew_subs 2024-03-29T11:12:04.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f691a1cb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:04.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f691a1cb640 1 -- 172.21.15.184:0/3810798893 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69141e3a50 con 0x7f691409ac40 2024-03-29T11:12:04.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.090+0000 7f691a1cb640 1 librados: init done 2024-03-29T11:12:04.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.091+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6904004bf0 con 0x7f691409ac40 2024-03-29T11:12:04.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.091+0000 7f68fbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:04.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.091+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f6904004d60 con 0x7f691409ac40 2024-03-29T11:12:04.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.091+0000 7f68fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:04.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.091+0000 7f68fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:04.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.091+0000 7f68fbfff640 10 monclient: dump: 2024-03-29T11:12:04.090 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2024-03-29T11:12:04.090 INFO:teuthology.orchestra.run.smithi184.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:04.091 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.091 INFO:teuthology.orchestra.run.smithi184.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.091 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:04.091 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2024-03-29T11:12:04.091 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:04.091 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:04.091 INFO:teuthology.orchestra.run.smithi184.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:04.091 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:12:04.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.091+0000 7f6911ffb640 4 set_mon_vals no callback set 2024-03-29T11:12:04.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.091+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6904005250 con 0x7f691409ac40 2024-03-29T11:12:04.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.091+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1246762396 0 0) 0x7f69040069a0 con 0x7f691409ac40 2024-03-29T11:12:04.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.091+0000 7f68fbfff640 10 client.4147.objecter ms_dispatch 0x7f6914000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:12:04.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.091+0000 7f68fbfff640 3 client.4147.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:12:04.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.091+0000 7f68fbfff640 3 client.4147.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:12:04.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.091+0000 7f68fbfff640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T11:12:04.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.091+0000 7f691a1cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:04.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.091+0000 7f691a1cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:04.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.091+0000 7f691a1cb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:04.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.091+0000 7f691a1cb640 1 -- 172.21.15.184:0/3810798893 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f4005080 con 0x7f691409ac40 2024-03-29T11:12:04.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.095+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6904006340 con 0x7f691409ac40 2024-03-29T11:12:04.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.095+0000 7f68fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:04.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.095+0000 7f68fbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:04.095 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:04.106+0000 7f1e7ea43040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:12:04.095 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:04.116+0000 7f84d7221040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:12:04.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.167+0000 7f691a1cb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "71f6869e-a468-4f2b-8f4b-c9c6005266b4", "id": 7}] 2024-03-29T11:12:04.146 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.167+0000 7f691a1cb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "71f6869e-a468-4f2b-8f4b-c9c6005266b4", "id": 7}] 2024-03-29T11:12:04.146 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.167+0000 7f691a1cb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:04.146 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.167+0000 7f691a1cb640 1 -- 172.21.15.184:0/3810798893 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd new", "uuid": "71f6869e-a468-4f2b-8f4b-c9c6005266b4", "id": 7} v 0) v1 -- 0x7f68f40058b0 con 0x7f691409ac40 2024-03-29T11:12:04.149 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.171+0000 7f68fbfff640 1 -- 172.21.15.184:0/3810798893 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "71f6869e-a468-4f2b-8f4b-c9c6005266b4", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 49024727 0 1371665406) 0x7f69040066b0 con 0x7f691409ac40 2024-03-29T11:12:04.149 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.171+0000 7f68fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "71f6869e-a468-4f2b-8f4b-c9c6005266b4", "id": 7}] 2024-03-29T11:12:04.149 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.171+0000 7f68fbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:04.149 INFO:teuthology.orchestra.run.smithi184.stdout:7 2024-03-29T11:12:04.150 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.171+0000 7f691a1cb640 10 librados: watch_flush enter 2024-03-29T11:12:04.151 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.171+0000 7f691a1cb640 10 librados: watch_flush exit 2024-03-29T11:12:04.151 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.171+0000 7f691a1cb640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T11:12:04.151 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.171+0000 7f691a1cb640 10 client.4147.objecter successfully canceled tick 2024-03-29T11:12:04.151 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.171+0000 7f691a1cb640 10 monclient: shutdown 2024-03-29T11:12:04.151 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.171+0000 7f691a1cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:04.151 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.171+0000 7f691a1cb640 1 -- 172.21.15.184:0/3810798893 >> v1:172.21.15.184:6789/0 conn(0x7f691409ac40 legacy=0x7f69141dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.151 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.171+0000 7f691a1cb640 1 -- 172.21.15.184:0/3810798893 shutdown_connections 2024-03-29T11:12:04.151 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.171+0000 7f691a1cb640 1 -- 172.21.15.184:0/3810798893 >> 172.21.15.184:0/3810798893 conn(0x7f6914066b00 msgr2=0x7f69140b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:04.151 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.172+0000 7f691a1cb640 1 -- 172.21.15.184:0/3810798893 shutdown_connections 2024-03-29T11:12:04.151 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.172+0000 7f691a1cb640 1 -- 172.21.15.184:0/3810798893 wait complete. 2024-03-29T11:12:04.151 INFO:teuthology.orchestra.run.smithi184.stderr:2024-03-29T11:12:04.172+0000 7f691a1cb640 1 librados: shutdown 2024-03-29T11:12:04.160 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:12:04.161 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:12:04.164 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:12:04.164 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:12:04.164 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:12:04.166 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:12:04.167 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:12:04.167 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:12:04.169 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:12:04.170 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:12:04.170 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:12:04.172 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:12:04.172 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:12:04.172 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:12:04.174 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:12:04.174 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:12:04.174 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:12:04.176 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:12:04.177 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:12:04.177 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:12:04.179 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:12:04.179 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:12:04.179 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:12:04.181 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:12:04.181 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:04.234 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:04.256+0000 7f1e7ea43040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:12:04.239 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:04.262+0000 7f84d7221040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:12:04.298 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:04.321+0000 7f84d7221040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:12:04.305 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.328+0000 7fdae6a9a640 10 monclient: get_monmap_and_config 2024-03-29T11:12:04.305 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.328+0000 7fdae6a9a640 10 monclient: build_initial_monmap 2024-03-29T11:12:04.306 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.328+0000 7fdae6a9a640 10 monclient: monmap: 2024-03-29T11:12:04.306 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2024-03-29T11:12:04.306 INFO:teuthology.orchestra.run.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:04.306 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2024-03-29T11:12:04.328793+0000 2024-03-29T11:12:04.306 INFO:teuthology.orchestra.run.smithi170.stderr:created 2024-03-29T11:12:04.328793+0000 2024-03-29T11:12:04.306 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:04.306 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:04.306 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:04.306 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:12:04.306 INFO:teuthology.orchestra.run.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:12:04.306 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-29T11:12:04.316 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.329+0000 7fdae6a9a640 1 Processor -- start 2024-03-29T11:12:04.316 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.329+0000 7fdae6a9a640 1 -- start start 2024-03-29T11:12:04.317 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.329+0000 7fdae6a9a640 10 monclient: init 2024-03-29T11:12:04.317 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.329+0000 7fdae6a9a640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:04.317 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.329+0000 7fdae6a9a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:12:04.317 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.329+0000 7fdae6a9a640 10 monclient(hunting): picked mon.noname-b con 0x7fdae009ac40 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:04.317 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.329+0000 7fdae6a9a640 10 monclient(hunting): picked mon.noname-a con 0x7fdae009c870 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:04.317 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.329+0000 7fdae6a9a640 10 monclient(hunting): picked mon.noname-c con 0x7fdae00b4360 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:04.317 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.329+0000 7fdae6a9a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae009dec0 con 0x7fdae009c870 2024-03-29T11:12:04.317 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.329+0000 7fdae6a9a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae009da30 con 0x7fdae00b4360 2024-03-29T11:12:04.317 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.329+0000 7fdae6a9a640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae00b39c0 con 0x7fdae009ac40 2024-03-29T11:12:04.317 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.329+0000 7fdadffff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fdae009c870 0x7fdae013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:35764/0 (socket says 172.21.15.170:35764) 2024-03-29T11:12:04.317 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.329+0000 7fdae6a9a640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:04.317 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.329+0000 7fdadffff640 1 -- 172.21.15.170:0/212815350 learned_addr learned my addr 172.21.15.170:0/212815350 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:12:04.317 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.329+0000 7fdae6a9a640 10 monclient(hunting): authenticate will time out at 905.640895s 2024-03-29T11:12:04.317 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 1 -- 172.21.15.170:0/212815350 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571626075 0 0) 0x7fdacc001040 con 0x7fdae009c870 2024-03-29T11:12:04.317 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 1 -- 172.21.15.170:0/212815350 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdad0003520 con 0x7fdae009c870 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 1 -- 172.21.15.170:0/212815350 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694210417 0 0) 0x7fdae00b39c0 con 0x7fdae009ac40 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 1 -- 172.21.15.170:0/212815350 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdacc001040 con 0x7fdae009ac40 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 1 -- 172.21.15.170:0/212815350 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740015600 0 0) 0x7fdae009da30 con 0x7fdae00b4360 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:04.319 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient(hunting): my global_id is 4174 2024-03-29T11:12:04.319 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 1 -- 172.21.15.170:0/212815350 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdae00b39c0 con 0x7fdae00b4360 2024-03-29T11:12:04.319 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 1 -- 172.21.15.170:0/212815350 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 365195840 0 0) 0x7fdad0003520 con 0x7fdae009c870 2024-03-29T11:12:04.319 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 1 -- 172.21.15.170:0/212815350 >> v1:172.21.15.170:6790/0 conn(0x7fdae009ac40 legacy=0x7fdae00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.319 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 1 -- 172.21.15.170:0/212815350 >> v1:172.21.15.184:6789/0 conn(0x7fdae00b4360 legacy=0x7fdae014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.319 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:12:04.319 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 1 monclient: found mon.noname-a 2024-03-29T11:12:04.319 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:04.319 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:12:04.319 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 1 -- 172.21.15.170:0/212815350 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae00b5160 con 0x7fdae009c870 2024-03-29T11:12:04.319 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:12:04.319 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:12:04.319 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:04.319 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 1 -- 172.21.15.170:0/212815350 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fdacc0015b0 con 0x7fdae009c870 2024-03-29T11:12:04.319 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdae6a9a640 5 monclient: authenticate success, global_id 4188 2024-03-29T11:12:04.320 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdae6a9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:04.320 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:04.320 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:04.320 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient: dump: 2024-03-29T11:12:04.320 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2024-03-29T11:12:04.320 INFO:teuthology.orchestra.run.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:04.320 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.320 INFO:teuthology.orchestra.run.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.320 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:04.320 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:04.320 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:04.320 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:04.320 INFO:teuthology.orchestra.run.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:04.320 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-29T11:12:04.320 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdae6a9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:04.321 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 1 -- 172.21.15.170:0/212815350 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdacc001710 con 0x7fdae009c870 2024-03-29T11:12:04.321 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:04.321 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 1 -- 172.21.15.170:0/212815350 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fdacc001b60 con 0x7fdae009c870 2024-03-29T11:12:04.321 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:04.321 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:04.321 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdadf7fe640 10 monclient: dump: 2024-03-29T11:12:04.321 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2024-03-29T11:12:04.321 INFO:teuthology.orchestra.run.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:04.321 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.321 INFO:teuthology.orchestra.run.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.321 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:04.321 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:04.321 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:04.321 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:04.321 INFO:teuthology.orchestra.run.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:04.321 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-29T11:12:04.322 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdae6a9a640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:04.322 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.330+0000 7fdae6a9a640 4 set_mon_vals no callback set 2024-03-29T11:12:04.322 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.331+0000 7fdae6a9a640 10 monclient: shutdown 2024-03-29T11:12:04.322 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.331+0000 7fdae6a9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:04.322 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.331+0000 7fdae6a9a640 1 -- 172.21.15.170:0/212815350 >> v1:172.21.15.170:6789/0 conn(0x7fdae009c870 legacy=0x7fdae013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.322 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.331+0000 7fdae6a9a640 1 -- 172.21.15.170:0/212815350 shutdown_connections 2024-03-29T11:12:04.322 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.331+0000 7fdae6a9a640 1 -- 172.21.15.170:0/212815350 >> 172.21.15.170:0/212815350 conn(0x7fdae0066b00 msgr2=0x7fdae0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:04.322 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.331+0000 7fdae6a9a640 1 -- 172.21.15.170:0/212815350 shutdown_connections 2024-03-29T11:12:04.322 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.331+0000 7fdae6a9a640 1 -- 172.21.15.170:0/212815350 wait complete. 2024-03-29T11:12:04.322 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.331+0000 7fdae6a9a640 10 monclient: build_initial_monmap 2024-03-29T11:12:04.322 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.331+0000 7fdae6a9a640 10 monclient: monmap: 2024-03-29T11:12:04.322 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2024-03-29T11:12:04.322 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:04.322 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2024-03-29T11:12:04.323 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2024-03-29T11:12:04.323 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:04.323 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:04.323 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:04.323 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:12:04.323 INFO:teuthology.orchestra.run.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:12:04.323 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-29T11:12:04.323 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.332+0000 7fdae6a9a640 1 librados: starting msgr at 2024-03-29T11:12:04.323 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.332+0000 7fdae6a9a640 1 librados: starting objecter 2024-03-29T11:12:04.323 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.332+0000 7fdae6a9a640 1 Processor -- start 2024-03-29T11:12:04.323 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.332+0000 7fdae6a9a640 1 -- start start 2024-03-29T11:12:04.323 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.332+0000 7fdae6a9a640 1 librados: setting wanted keys 2024-03-29T11:12:04.323 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.332+0000 7fdae6a9a640 1 librados: calling monclient init 2024-03-29T11:12:04.323 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.332+0000 7fdae6a9a640 10 monclient: init 2024-03-29T11:12:04.323 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdae6a9a640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:04.324 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdae6a9a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:04.324 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdae6a9a640 10 monclient(hunting): picked mon.noname-a con 0x7fdae009ac40 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:04.324 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdae6a9a640 10 monclient(hunting): picked mon.noname-c con 0x7fdae0140b60 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:04.324 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdae6a9a640 10 monclient(hunting): picked mon.noname-b con 0x7fdae0141d60 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:04.324 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdae4836640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fdae009ac40 0x7fdae01dabd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:35778/0 (socket says 172.21.15.170:35778) 2024-03-29T11:12:04.324 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdae4836640 1 -- 172.21.15.170:0/1274720787 learned_addr learned my addr 172.21.15.170:0/1274720787 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:12:04.324 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadffff640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fdae0140b60 0x7fdae0141650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:53842/0 (socket says 172.21.15.170:53842) 2024-03-29T11:12:04.324 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdae6a9a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae0140010 con 0x7fdae009ac40 2024-03-29T11:12:04.324 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdae6a9a640 1 -- 172.21.15.170:0/1274720787 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae00afbc0 con 0x7fdae0141d60 2024-03-29T11:12:04.324 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 client.?.objecter ms_handle_connect 0x7fdae009ac40 2024-03-29T11:12:04.324 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:04.324 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 client.?.objecter ms_handle_connect 0x7fdae0140b60 2024-03-29T11:12:04.324 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:04.324 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdae6a9a640 1 -- 172.21.15.170:0/1274720787 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae00afd50 con 0x7fdae0140b60 2024-03-29T11:12:04.324 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380809886 0 0) 0x7fdae0140010 con 0x7fdae009ac40 2024-03-29T11:12:04.325 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdae6a9a640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:04.325 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdae6a9a640 10 monclient(hunting): authenticate will time out at 905.644401s 2024-03-29T11:12:04.325 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:12:04.325 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:04.325 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:04.325 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:12:04.325 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdac4004eb0 con 0x7fdae009ac40 2024-03-29T11:12:04.325 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 client.?.objecter ms_handle_connect 0x7fdae0141d60 2024-03-29T11:12:04.325 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:04.325 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194420913 0 0) 0x7fdae00afd50 con 0x7fdae0140b60 2024-03-29T11:12:04.325 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:12:04.325 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:04.325 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:04.325 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:12:04.325 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdae0140010 con 0x7fdae0140b60 2024-03-29T11:12:04.326 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407842371 0 0) 0x7fdae00afbc0 con 0x7fdae0141d60 2024-03-29T11:12:04.326 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:12:04.326 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:04.326 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:04.326 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient(hunting): my global_id is 4177 2024-03-29T11:12:04.326 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdae00afd50 con 0x7fdae0141d60 2024-03-29T11:12:04.326 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 983358176 0 0) 0x7fdac4004eb0 con 0x7fdae009ac40 2024-03-29T11:12:04.326 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdae00afbc0 con 0x7fdae009ac40 2024-03-29T11:12:04.326 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fdad4001a50 con 0x7fdae009ac40 2024-03-29T11:12:04.326 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:04.326 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:04.326 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient(hunting): dump: 2024-03-29T11:12:04.326 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2024-03-29T11:12:04.326 INFO:teuthology.orchestra.run.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:04.327 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.327 INFO:teuthology.orchestra.run.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.327 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:04.327 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:04.327 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:04.327 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:04.327 INFO:teuthology.orchestra.run.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:04.327 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-29T11:12:04.327 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:04.327 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4109482365 0 0) 0x7fdae00afd50 con 0x7fdae0141d60 2024-03-29T11:12:04.327 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdac4004eb0 con 0x7fdae0141d60 2024-03-29T11:12:04.327 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fdad8001a30 con 0x7fdae0141d60 2024-03-29T11:12:04.327 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:04.327 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:04.327 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient(hunting): dump: 2024-03-29T11:12:04.328 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2024-03-29T11:12:04.328 INFO:teuthology.orchestra.run.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:04.328 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.328 INFO:teuthology.orchestra.run.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.328 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:04.328 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:04.328 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:04.328 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:04.328 INFO:teuthology.orchestra.run.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:04.328 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-29T11:12:04.328 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1844978182 0 0) 0x7fdae00afbc0 con 0x7fdae009ac40 2024-03-29T11:12:04.328 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 >> v1:172.21.15.170:6790/0 conn(0x7fdae0140b60 legacy=0x7fdae0141650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.328 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 >> v1:172.21.15.184:6789/0 conn(0x7fdae0141d60 legacy=0x7fdae01e1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.328 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:12:04.328 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 1 monclient: found mon.a 2024-03-29T11:12:04.329 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:04.329 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:04.329 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae01db2e0 con 0x7fdae009ac40 2024-03-29T11:12:04.329 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:12:04.329 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:12:04.329 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdadd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:04.329 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdae6a9a640 5 monclient: authenticate success, global_id 4191 2024-03-29T11:12:04.329 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdae6a9a640 10 monclient: _renew_subs 2024-03-29T11:12:04.329 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdae6a9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:04.329 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdae6a9a640 1 -- 172.21.15.170:0/1274720787 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae01e3240 con 0x7fdae009ac40 2024-03-29T11:12:04.329 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.333+0000 7fdae6a9a640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:04.329 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdad4001d10 con 0x7fdae009ac40 2024-03-29T11:12:04.329 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdadd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:04.329 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdae6a9a640 10 monclient: _renew_subs 2024-03-29T11:12:04.329 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdadeffd640 4 set_mon_vals no callback set 2024-03-29T11:12:04.330 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdae6a9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:04.330 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdae6a9a640 1 -- 172.21.15.170:0/1274720787 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae01e35b0 con 0x7fdae009ac40 2024-03-29T11:12:04.330 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdae6a9a640 1 librados: init done 2024-03-29T11:12:04.330 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fdad4003a40 con 0x7fdae009ac40 2024-03-29T11:12:04.330 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdadd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:04.330 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdadd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:04.330 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdadd7fa640 10 monclient: dump: 2024-03-29T11:12:04.330 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2024-03-29T11:12:04.330 INFO:teuthology.orchestra.run.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:04.330 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.330 INFO:teuthology.orchestra.run.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:04.330 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:04.330 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:04.330 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:04.330 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:04.331 INFO:teuthology.orchestra.run.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:04.331 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-29T11:12:04.331 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdad4003fb0 con 0x7fdae009ac40 2024-03-29T11:12:04.331 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fdad4005c90 con 0x7fdae009ac40 2024-03-29T11:12:04.331 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdadd7fa640 10 client.4191.objecter ms_dispatch 0x7fdae0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:04.331 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdadd7fa640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:04.331 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdadd7fa640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:04.331 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdadd7fa640 20 client.4191.objecter dump_active .. 0 homeless 2024-03-29T11:12:04.331 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdae6a9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:04.331 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdae6a9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:04.331 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdae6a9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:04.331 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.334+0000 7fdae6a9a640 1 -- 172.21.15.170:0/1274720787 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdae01da870 con 0x7fdae009ac40 2024-03-29T11:12:04.331 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.336+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdad40038a0 con 0x7fdae009ac40 2024-03-29T11:12:04.331 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.336+0000 7fdadd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:04.331 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.336+0000 7fdadd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:04.366 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:04.388+0000 7f1e7ea43040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:12:04.375 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.397+0000 7fdae6a9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:04.375 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.397+0000 7fdae6a9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:04.375 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.397+0000 7fdae6a9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:04.375 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.397+0000 7fdae6a9a640 1 -- 172.21.15.170:0/1274720787 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdae004f730 con 0x7fdae009ac40 2024-03-29T11:12:04.375 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.398+0000 7fdadd7fa640 1 -- 172.21.15.170:0/1274720787 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fdad4004260 con 0x7fdae009ac40 2024-03-29T11:12:04.375 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.398+0000 7fdadd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:04.376 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.398+0000 7fdadd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:04.376 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-29T11:12:04.376 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:04.377 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.399+0000 7fdae6a9a640 10 librados: watch_flush enter 2024-03-29T11:12:04.377 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.399+0000 7fdae6a9a640 10 librados: watch_flush exit 2024-03-29T11:12:04.377 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.399+0000 7fdae6a9a640 20 client.4191.objecter shutdown clearing up homeless session... 2024-03-29T11:12:04.377 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.399+0000 7fdae6a9a640 10 client.4191.objecter successfully canceled tick 2024-03-29T11:12:04.378 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.399+0000 7fdae6a9a640 10 monclient: shutdown 2024-03-29T11:12:04.378 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.399+0000 7fdae6a9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:04.378 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.399+0000 7fdae6a9a640 1 -- 172.21.15.170:0/1274720787 >> v1:172.21.15.170:6789/0 conn(0x7fdae009ac40 legacy=0x7fdae01dabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.378 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.399+0000 7fdae6a9a640 1 -- 172.21.15.170:0/1274720787 shutdown_connections 2024-03-29T11:12:04.378 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.399+0000 7fdae6a9a640 1 -- 172.21.15.170:0/1274720787 >> 172.21.15.170:0/1274720787 conn(0x7fdae0066b00 msgr2=0x7fdae013a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:04.378 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.399+0000 7fdae6a9a640 1 -- 172.21.15.170:0/1274720787 shutdown_connections 2024-03-29T11:12:04.378 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.399+0000 7fdae6a9a640 1 -- 172.21.15.170:0/1274720787 wait complete. 2024-03-29T11:12:04.378 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-29T11:12:04.399+0000 7fdae6a9a640 1 librados: shutdown 2024-03-29T11:12:04.387 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:12:04.387 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:12:04.411 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:04.432+0000 7f1e7ea43040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:12:04.417 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:04.440+0000 7f84d7221040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:12:04.512 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:04.534+0000 7f84d7221040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:12:04.522 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:04.543+0000 7f1e7ea43040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:12:04.619 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:04.640+0000 7f1e7ea43040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:12:04.666 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:04.688+0000 7f84d7221040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:12:04.707 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:04.730+0000 7f84d7221040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:12:04.766 INFO:tasks.ceph.osd.1.smithi170.stderr:2024-03-29T11:12:04.788+0000 7f8a9781e640 -1 Falling back to public interface 2024-03-29T11:12:04.770 INFO:tasks.ceph.osd.4.smithi184.stderr:2024-03-29T11:12:04.792+0000 7f32fc9e1640 -1 Falling back to public interface 2024-03-29T11:12:04.771 INFO:tasks.ceph.osd.6.smithi184.stderr:2024-03-29T11:12:04.793+0000 7f672ea68640 -1 Falling back to public interface 2024-03-29T11:12:04.772 INFO:tasks.ceph.osd.0.smithi170.stderr:2024-03-29T11:12:04.794+0000 7fcabd613640 -1 Falling back to public interface 2024-03-29T11:12:04.772 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:04.793+0000 7f1e7ea43040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:12:04.794 INFO:tasks.ceph.osd.2.smithi170.stderr:2024-03-29T11:12:04.816+0000 7fb5ec3df640 -1 Falling back to public interface 2024-03-29T11:12:04.795 INFO:tasks.ceph.osd.3.smithi170.stderr:2024-03-29T11:12:04.817+0000 7f3aedd06640 -1 Falling back to public interface 2024-03-29T11:12:04.804 INFO:tasks.ceph.osd.7.smithi184.stderr:2024-03-29T11:12:04.825+0000 7f1099c0e640 -1 Falling back to public interface 2024-03-29T11:12:04.809 INFO:tasks.ceph.osd.5.smithi184.stderr:2024-03-29T11:12:04.830+0000 7fa021813640 -1 Falling back to public interface 2024-03-29T11:12:04.816 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:04.838+0000 7f1e7ea43040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:12:04.862 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:04.885+0000 7f84d7221040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:12:04.969 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:04.991+0000 7f1e7ea43040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:12:05.193 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:05.283 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.305+0000 7f9cd630e640 10 monclient: get_monmap_and_config 2024-03-29T11:12:05.283 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.305+0000 7f9cd630e640 10 monclient: build_initial_monmap 2024-03-29T11:12:05.283 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.305+0000 7f9cd630e640 10 monclient: monmap: 2024-03-29T11:12:05.283 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:12:05.283 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:05.283 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:12:05.306380+0000 2024-03-29T11:12:05.284 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:12:05.306380+0000 2024-03-29T11:12:05.284 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:05.284 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:05.284 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:05.284 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:12:05.284 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:12:05.284 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:05.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.306+0000 7f9cd630e640 1 Processor -- start 2024-03-29T11:12:05.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.306+0000 7f9cd630e640 1 -- start start 2024-03-29T11:12:05.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.306+0000 7f9cd630e640 10 monclient: init 2024-03-29T11:12:05.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cd630e640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:05.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cd630e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:12:05.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cd630e640 10 monclient(hunting): picked mon.noname-c con 0x7f9cd01457f0 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:05.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cd630e640 10 monclient(hunting): picked mon.noname-b con 0x7f9cd013f650 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:05.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cd630e640 10 monclient(hunting): picked mon.noname-a con 0x7f9cd00a9d70 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:05.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cd630e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd00aaef0 con 0x7f9cd00a9d70 2024-03-29T11:12:05.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cd630e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd0145bd0 con 0x7f9cd01457f0 2024-03-29T11:12:05.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cd630e640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd0145eb0 con 0x7f9cd013f650 2024-03-29T11:12:05.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cd630e640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:05.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cd630e640 10 monclient(hunting): authenticate will time out at 906.618424s 2024-03-29T11:12:05.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cd48ab640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f9cd00a9d70 0x7f9cd014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:35788/0 (socket says 172.21.15.170:35788) 2024-03-29T11:12:05.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9ccf7fe640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f9cd013f650 0x7f9cd00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:53848/0 (socket says 172.21.15.170:53848) 2024-03-29T11:12:05.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cd48ab640 1 -- 172.21.15.170:0/3792930252 learned_addr learned my addr 172.21.15.170:0/3792930252 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:12:05.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cceffd640 1 -- 172.21.15.170:0/3792930252 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484625057 0 0) 0x7f9cd0145eb0 con 0x7f9cd013f650 2024-03-29T11:12:05.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:05.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:05.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:05.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cceffd640 10 monclient(hunting): my global_id is 4193 2024-03-29T11:12:05.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cceffd640 1 -- 172.21.15.170:0/3792930252 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cb8003520 con 0x7f9cd013f650 2024-03-29T11:12:05.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cceffd640 1 -- 172.21.15.170:0/3792930252 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301643241 0 0) 0x7f9cd00aaef0 con 0x7f9cd00a9d70 2024-03-29T11:12:05.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:05.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:05.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:05.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cceffd640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:12:05.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cceffd640 1 -- 172.21.15.170:0/3792930252 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cd0145eb0 con 0x7f9cd00a9d70 2024-03-29T11:12:05.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cceffd640 1 -- 172.21.15.170:0/3792930252 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 479045921 0 0) 0x7f9cd0145bd0 con 0x7f9cd01457f0 2024-03-29T11:12:05.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:05.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:05.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:05.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cceffd640 10 monclient(hunting): my global_id is 4180 2024-03-29T11:12:05.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.307+0000 7f9cceffd640 1 -- 172.21.15.170:0/3792930252 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cd00aaef0 con 0x7f9cd01457f0 2024-03-29T11:12:05.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 1 -- 172.21.15.170:0/3792930252 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4189018676 0 0) 0x7f9cb8003520 con 0x7f9cd013f650 2024-03-29T11:12:05.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 1 -- 172.21.15.170:0/3792930252 >> v1:172.21.15.184:6789/0 conn(0x7f9cd01457f0 legacy=0x7f9cd013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 1 -- 172.21.15.170:0/3792930252 >> v1:172.21.15.170:6789/0 conn(0x7f9cd00a9d70 legacy=0x7f9cd014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:05.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 1 monclient: found mon.noname-b 2024-03-29T11:12:05.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:05.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:12:05.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 1 -- 172.21.15.170:0/3792930252 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd0146340 con 0x7f9cd013f650 2024-03-29T11:12:05.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 10 monclient: _finish_auth 0 2024-03-29T11:12:05.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:05.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:05.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 1 -- 172.21.15.170:0/3792930252 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9cc00015c0 con 0x7f9cd013f650 2024-03-29T11:12:05.298 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cd630e640 5 monclient: authenticate success, global_id 4193 2024-03-29T11:12:05.298 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cd630e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:05.298 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:05.298 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:05.298 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 10 monclient: dump: 2024-03-29T11:12:05.298 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:05.298 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:05.298 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:05.298 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:05.298 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:05.298 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:05.298 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:05.298 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:05.298 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:05.299 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:05.299 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cd630e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:05.299 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 1 -- 172.21.15.170:0/3792930252 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9cc00017a0 con 0x7f9cd013f650 2024-03-29T11:12:05.299 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:05.299 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 1 -- 172.21.15.170:0/3792930252 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9cc0001bf0 con 0x7f9cd013f650 2024-03-29T11:12:05.299 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cd630e640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:05.299 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cd630e640 4 set_mon_vals no callback set 2024-03-29T11:12:05.299 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:05.299 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:05.299 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cceffd640 10 monclient: dump: 2024-03-29T11:12:05.299 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:05.299 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:05.299 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:05.299 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:05.299 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:05.300 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:05.300 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:05.300 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:05.300 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:05.300 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:05.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cd630e640 10 monclient: shutdown 2024-03-29T11:12:05.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cd630e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:05.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cd630e640 1 -- 172.21.15.170:0/3792930252 >> v1:172.21.15.170:6790/0 conn(0x7f9cd013f650 legacy=0x7f9cd00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cd630e640 1 -- 172.21.15.170:0/3792930252 shutdown_connections 2024-03-29T11:12:05.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cd630e640 1 -- 172.21.15.170:0/3792930252 >> 172.21.15.170:0/3792930252 conn(0x7f9cd0066a00 msgr2=0x7f9cd0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:05.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cd630e640 1 -- 172.21.15.170:0/3792930252 shutdown_connections 2024-03-29T11:12:05.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.308+0000 7f9cd630e640 1 -- 172.21.15.170:0/3792930252 wait complete. 2024-03-29T11:12:05.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.309+0000 7f9cd630e640 10 monclient: build_initial_monmap 2024-03-29T11:12:05.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.309+0000 7f9cd630e640 10 monclient: monmap: 2024-03-29T11:12:05.301 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:12:05.301 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:05.301 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:12:05.301 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:12:05.301 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:05.301 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:05.301 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:05.301 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:12:05.301 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:12:05.301 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:05.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.309+0000 7f9cd630e640 1 librados: starting msgr at 2024-03-29T11:12:05.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.309+0000 7f9cd630e640 1 librados: starting objecter 2024-03-29T11:12:05.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.309+0000 7f9cd630e640 1 Processor -- start 2024-03-29T11:12:05.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cd630e640 1 -- start start 2024-03-29T11:12:05.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cd630e640 1 librados: setting wanted keys 2024-03-29T11:12:05.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cd630e640 1 librados: calling monclient init 2024-03-29T11:12:05.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cd630e640 10 monclient: init 2024-03-29T11:12:05.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cd630e640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:05.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cd630e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:05.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cd630e640 10 monclient(hunting): picked mon.noname-a con 0x7f9cd00a9d70 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:05.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cd630e640 10 monclient(hunting): picked mon.noname-c con 0x7f9cd013f650 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:05.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cd630e640 10 monclient(hunting): picked mon.noname-b con 0x7f9cd01457f0 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:05.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cd630e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd0141c80 con 0x7f9cd00a9d70 2024-03-29T11:12:05.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cd630e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd0141e10 con 0x7f9cd01457f0 2024-03-29T11:12:05.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cd630e640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd0141fa0 con 0x7f9cd013f650 2024-03-29T11:12:05.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cd630e640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:05.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cd630e640 10 monclient(hunting): authenticate will time out at 906.621657s 2024-03-29T11:12:05.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9ccffff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f9cd00a9d70 0x7f9cd013aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:35790/0 (socket says 172.21.15.170:35790) 2024-03-29T11:12:05.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9ccffff640 1 -- 172.21.15.170:0/3568851502 learned_addr learned my addr 172.21.15.170:0/3568851502 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:12:05.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cccff9640 10 client.?.objecter ms_handle_connect 0x7f9cd013f650 2024-03-29T11:12:05.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cccff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:05.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cccff9640 10 client.?.objecter ms_handle_connect 0x7f9cd00a9d70 2024-03-29T11:12:05.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cccff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:05.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cccff9640 10 client.?.objecter ms_handle_connect 0x7f9cd01457f0 2024-03-29T11:12:05.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cccff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:05.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435686662 0 0) 0x7f9cd0141fa0 con 0x7f9cd013f650 2024-03-29T11:12:05.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cccff9640 10 monclient(hunting): _negotiate 2024-03-29T11:12:05.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cccff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:05.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:05.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.310+0000 7f9cccff9640 10 monclient(hunting): my global_id is 4196 2024-03-29T11:12:05.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cbc004eb0 con 0x7f9cd013f650 2024-03-29T11:12:05.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881105905 0 0) 0x7f9cd0141c80 con 0x7f9cd00a9d70 2024-03-29T11:12:05.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient(hunting): _negotiate 2024-03-29T11:12:05.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:05.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:05.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient(hunting): my global_id is 4197 2024-03-29T11:12:05.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cd0141fa0 con 0x7f9cd00a9d70 2024-03-29T11:12:05.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025645705 0 0) 0x7f9cd0141e10 con 0x7f9cd01457f0 2024-03-29T11:12:05.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient(hunting): _negotiate 2024-03-29T11:12:05.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:05.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:05.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient(hunting): my global_id is 4183 2024-03-29T11:12:05.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cd0141c80 con 0x7f9cd01457f0 2024-03-29T11:12:05.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 637396947 0 0) 0x7f9cd0141c80 con 0x7f9cd01457f0 2024-03-29T11:12:05.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9cd0141e10 con 0x7f9cd01457f0 2024-03-29T11:12:05.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 743377066 0 0) 0x7f9cd0141fa0 con 0x7f9cd00a9d70 2024-03-29T11:12:05.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9cd0141c80 con 0x7f9cd00a9d70 2024-03-29T11:12:05.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3283318568 0 0) 0x7f9cbc004eb0 con 0x7f9cd013f650 2024-03-29T11:12:05.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9cd0141fa0 con 0x7f9cd013f650 2024-03-29T11:12:05.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9cc8001410 con 0x7f9cd01457f0 2024-03-29T11:12:05.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:05.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:05.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient(hunting): dump: 2024-03-29T11:12:05.305 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:05.305 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:05.305 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:05.305 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:05.305 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:05.305 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:05.305 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:05.305 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:05.305 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:05.305 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:05.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:05.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9cc40018d0 con 0x7f9cd00a9d70 2024-03-29T11:12:05.306 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:05.306 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:05.306 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient(hunting): dump: 2024-03-29T11:12:05.306 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:05.306 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:05.306 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:05.306 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:05.306 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:05.306 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:05.306 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:05.306 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:05.306 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:05.306 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:05.306 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9cc0001af0 con 0x7f9cd013f650 2024-03-29T11:12:05.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:05.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:05.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient(hunting): dump: 2024-03-29T11:12:05.307 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:05.307 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:05.307 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:05.307 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:05.307 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:05.307 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:05.307 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:05.307 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:05.307 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:05.307 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:05.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1791887928 0 0) 0x7f9cd0141e10 con 0x7f9cd01457f0 2024-03-29T11:12:05.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 >> v1:172.21.15.170:6790/0 conn(0x7f9cd013f650 legacy=0x7f9cd00b0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 >> v1:172.21.15.170:6789/0 conn(0x7f9cd00a9d70 legacy=0x7f9cd013aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient: _finish_hunting 0 2024-03-29T11:12:05.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 monclient: found mon.b 2024-03-29T11:12:05.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:05.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:05.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd00b1270 con 0x7f9cd01457f0 2024-03-29T11:12:05.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient: _finish_auth 0 2024-03-29T11:12:05.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient: _check_auth_tickets 2024-03-29T11:12:05.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:05.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cd630e640 5 monclient: authenticate success, global_id 4183 2024-03-29T11:12:05.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cd630e640 10 monclient: _renew_subs 2024-03-29T11:12:05.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cd630e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:05.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cd630e640 1 -- 172.21.15.170:0/3568851502 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd00b1520 con 0x7f9cd01457f0 2024-03-29T11:12:05.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cd630e640 10 client.4183.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:05.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cd630e640 10 monclient: _renew_subs 2024-03-29T11:12:05.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cd630e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:05.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cd630e640 1 -- 172.21.15.170:0/3568851502 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd0142450 con 0x7f9cd01457f0 2024-03-29T11:12:05.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cd630e640 1 librados: init done 2024-03-29T11:12:05.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9cc8001960 con 0x7f9cd01457f0 2024-03-29T11:12:05.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:05.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9cc8004d00 con 0x7f9cd01457f0 2024-03-29T11:12:05.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:05.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:05.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 monclient: dump: 2024-03-29T11:12:05.309 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:05.309 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:05.309 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:05.309 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:05.309 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:05.310 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:05.310 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:05.310 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:05.310 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:05.310 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:05.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cce7fc640 4 set_mon_vals no callback set 2024-03-29T11:12:05.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9cc80051f0 con 0x7f9cd01457f0 2024-03-29T11:12:05.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f9cc8005b70 con 0x7f9cd01457f0 2024-03-29T11:12:05.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 10 client.4183.objecter ms_dispatch 0x7f9cd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:05.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 3 client.4183.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:05.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 3 client.4183.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:05.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.311+0000 7f9cccff9640 20 client.4183.objecter dump_active .. 0 homeless 2024-03-29T11:12:05.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.312+0000 7f9cd630e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:05.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.312+0000 7f9cd630e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:05.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.312+0000 7f9cd630e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:05.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.312+0000 7f9cd630e640 1 -- 172.21.15.170:0/3568851502 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd0075f10 con 0x7f9cd01457f0 2024-03-29T11:12:05.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.315+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9cc8005f10 con 0x7f9cd01457f0 2024-03-29T11:12:05.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.315+0000 7f9cccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:05.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.315+0000 7f9cccff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:05.354 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.376+0000 7f9cd630e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:05.354 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.376+0000 7f9cd630e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:05.354 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.376+0000 7f9cd630e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:05.354 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.376+0000 7f9cd630e640 1 -- 172.21.15.170:0/3568851502 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9cd00b3680 con 0x7f9cd01457f0 2024-03-29T11:12:05.355 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.377+0000 7f9cccff9640 1 -- 172.21.15.170:0/3568851502 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f9cc8005550 con 0x7f9cd01457f0 2024-03-29T11:12:05.355 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.377+0000 7f9cccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:05.355 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.377+0000 7f9cccff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:05.355 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:12:05.356 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:05.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.378+0000 7f9cd630e640 10 librados: watch_flush enter 2024-03-29T11:12:05.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.378+0000 7f9cd630e640 10 librados: watch_flush exit 2024-03-29T11:12:05.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.378+0000 7f9cd630e640 20 client.4183.objecter shutdown clearing up homeless session... 2024-03-29T11:12:05.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.378+0000 7f9cd630e640 10 client.4183.objecter successfully canceled tick 2024-03-29T11:12:05.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.378+0000 7f9cd630e640 10 monclient: shutdown 2024-03-29T11:12:05.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.378+0000 7f9cd630e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:05.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.378+0000 7f9cd630e640 1 -- 172.21.15.170:0/3568851502 >> v1:172.21.15.184:6789/0 conn(0x7f9cd01457f0 legacy=0x7f9cd01d70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.378+0000 7f9cd630e640 1 -- 172.21.15.170:0/3568851502 shutdown_connections 2024-03-29T11:12:05.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.378+0000 7f9cd630e640 1 -- 172.21.15.170:0/3568851502 >> 172.21.15.170:0/3568851502 conn(0x7f9cd0066a00 msgr2=0x7f9cd00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:05.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.378+0000 7f9cd630e640 1 -- 172.21.15.170:0/3568851502 shutdown_connections 2024-03-29T11:12:05.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.378+0000 7f9cd630e640 1 -- 172.21.15.170:0/3568851502 wait complete. 2024-03-29T11:12:05.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:05.378+0000 7f9cd630e640 1 librados: shutdown 2024-03-29T11:12:05.365 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:05.669 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:05.692+0000 7f84d7221040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:12:05.793 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:05.814+0000 7f1e7ea43040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:12:05.950 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:05.966+0000 7f84d7221040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:12:05.995 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:06.018+0000 7f84d7221040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:12:06.066 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:06.088+0000 7f1e7ea43040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:12:06.111 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:06.130+0000 7f84d7221040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:12:06.118 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:06.139+0000 7f1e7ea43040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:12:06.235 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-03-29T11:12:06.257+0000 7f1e7ea43040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:12:07.122 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:07.143+0000 7f8477a67640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:12:07.122 INFO:tasks.ceph.mgr.y.smithi170.stderr:2024-03-29T11:12:07.144+0000 7f8477a67640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:12:08.239 INFO:tasks.ceph.osd.6.smithi184.stderr:2024-03-29T11:12:08.261+0000 7f672ea68640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:12:08.263 INFO:tasks.ceph.osd.1.smithi170.stderr:2024-03-29T11:12:08.285+0000 7f8a9781e640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:12:08.287 INFO:tasks.ceph.osd.4.smithi184.stderr:2024-03-29T11:12:08.309+0000 7f32fc9e1640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:12:08.304 INFO:tasks.ceph.osd.0.smithi170.stderr:2024-03-29T11:12:08.325+0000 7fcabd613640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:12:08.335 INFO:tasks.ceph.osd.7.smithi184.stderr:2024-03-29T11:12:08.357+0000 7f1099c0e640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:12:08.340 INFO:tasks.ceph.osd.5.smithi184.stderr:2024-03-29T11:12:08.362+0000 7fa021813640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:12:08.341 INFO:tasks.ceph.osd.3.smithi170.stderr:2024-03-29T11:12:08.362+0000 7f3aedd06640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:12:08.353 INFO:tasks.ceph.osd.2.smithi170.stderr:2024-03-29T11:12:08.375+0000 7fb5ec3df640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:12:12.172 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:12.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.359+0000 7fb2ac415640 10 monclient: get_monmap_and_config 2024-03-29T11:12:12.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.359+0000 7fb2ac415640 10 monclient: build_initial_monmap 2024-03-29T11:12:12.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.359+0000 7fb2ac415640 10 monclient: monmap: 2024-03-29T11:12:12.338 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:12:12.338 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:12.338 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:12:12.361007+0000 2024-03-29T11:12:12.338 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:12:12.361007+0000 2024-03-29T11:12:12.338 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:12.338 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:12.339 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:12.339 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:12:12.339 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:12:12.339 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:12.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.360+0000 7fb2ac415640 1 Processor -- start 2024-03-29T11:12:12.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.360+0000 7fb2ac415640 1 -- start start 2024-03-29T11:12:12.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.360+0000 7fb2ac415640 10 monclient: init 2024-03-29T11:12:12.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.361+0000 7fb2ac415640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:12.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.361+0000 7fb2ac415640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:12:12.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.361+0000 7fb2ac415640 10 monclient(hunting): picked mon.noname-c con 0x7fb2a4071de0 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:12.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.361+0000 7fb2ac415640 10 monclient(hunting): picked mon.noname-b con 0x7fb2a40721c0 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:12.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.361+0000 7fb2ac415640 10 monclient(hunting): picked mon.noname-a con 0x7fb2a4145ae0 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:12.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.361+0000 7fb2ac415640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a4145650 con 0x7fb2a4145ae0 2024-03-29T11:12:12.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.361+0000 7fb2ac415640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a40ae260 con 0x7fb2a4071de0 2024-03-29T11:12:12.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.361+0000 7fb2ac415640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a41420d0 con 0x7fb2a40721c0 2024-03-29T11:12:12.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.361+0000 7fb2ac415640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:12.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.361+0000 7fb2ac415640 10 monclient(hunting): authenticate will time out at 913.673803s 2024-03-29T11:12:12.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.362+0000 7fb2a99b0640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fb2a40721c0 0x7fb2a413f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:40938/0 (socket says 172.21.15.170:40938) 2024-03-29T11:12:12.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.362+0000 7fb2a99b0640 1 -- 172.21.15.170:0/1999303920 learned_addr learned my addr 172.21.15.170:0/1999303920 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:12:12.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.362+0000 7fb2a91af640 1 -- 172.21.15.170:0/1999303920 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895860009 0 0) 0x7fb2a40ae260 con 0x7fb2a4071de0 2024-03-29T11:12:12.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.362+0000 7fb2a91af640 10 monclient(hunting): _negotiate 2024-03-29T11:12:12.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.362+0000 7fb2a91af640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:12.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.362+0000 7fb2a91af640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:12.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.362+0000 7fb2a91af640 10 monclient(hunting): my global_id is 4228 2024-03-29T11:12:12.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.362+0000 7fb2a91af640 1 -- 172.21.15.170:0/1999303920 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb290003520 con 0x7fb2a4071de0 2024-03-29T11:12:12.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.362+0000 7fb2a91af640 1 -- 172.21.15.170:0/1999303920 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913857996 0 0) 0x7fb2a4145650 con 0x7fb2a4145ae0 2024-03-29T11:12:12.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.362+0000 7fb2a91af640 10 monclient(hunting): _negotiate 2024-03-29T11:12:12.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.362+0000 7fb2a91af640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:12.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.362+0000 7fb2a91af640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:12.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.362+0000 7fb2a91af640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:12:12.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.362+0000 7fb2a91af640 1 -- 172.21.15.170:0/1999303920 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a40ae260 con 0x7fb2a4145ae0 2024-03-29T11:12:12.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 1 -- 172.21.15.170:0/1999303920 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789978658 0 0) 0x7fb2a41420d0 con 0x7fb2a40721c0 2024-03-29T11:12:12.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 10 monclient(hunting): _negotiate 2024-03-29T11:12:12.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:12.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:12.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:12:12.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 1 -- 172.21.15.170:0/1999303920 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a4145650 con 0x7fb2a40721c0 2024-03-29T11:12:12.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 1 -- 172.21.15.170:0/1999303920 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3519371888 0 0) 0x7fb290003520 con 0x7fb2a4071de0 2024-03-29T11:12:12.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 1 -- 172.21.15.170:0/1999303920 >> v1:172.21.15.170:6790/0 conn(0x7fb2a40721c0 legacy=0x7fb2a413f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:12.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 1 -- 172.21.15.170:0/1999303920 >> v1:172.21.15.170:6789/0 conn(0x7fb2a4145ae0 legacy=0x7fb2a4145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:12.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 10 monclient: _finish_hunting 0 2024-03-29T11:12:12.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 1 monclient: found mon.noname-c 2024-03-29T11:12:12.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:12.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:12:12.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 1 -- 172.21.15.170:0/1999303920 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a40b0590 con 0x7fb2a4071de0 2024-03-29T11:12:12.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 10 monclient: _finish_auth 0 2024-03-29T11:12:12.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 10 monclient: _check_auth_tickets 2024-03-29T11:12:12.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:12.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2ac415640 5 monclient: authenticate success, global_id 4228 2024-03-29T11:12:12.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2ac415640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:12.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 1 -- 172.21.15.170:0/1999303920 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb2a0001690 con 0x7fb2a4071de0 2024-03-29T11:12:12.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:12.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:12.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2a91af640 10 monclient: dump: 2024-03-29T11:12:12.348 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:12.348 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:12.348 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:12.348 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:12.348 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:12.348 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:12.348 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:12.348 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:12.348 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:12.348 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:12.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.363+0000 7fb2ac415640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:12.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.364+0000 7fb2a91af640 1 -- 172.21.15.170:0/1999303920 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2a0001870 con 0x7fb2a4071de0 2024-03-29T11:12:12.349 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.364+0000 7fb2a91af640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:12.349 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.364+0000 7fb2a91af640 1 -- 172.21.15.170:0/1999303920 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb2a0001bc0 con 0x7fb2a4071de0 2024-03-29T11:12:12.349 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.364+0000 7fb2a91af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:12.349 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.364+0000 7fb2a91af640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:12.349 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.364+0000 7fb2a91af640 10 monclient: dump: 2024-03-29T11:12:12.349 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:12.349 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:12.349 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:12.349 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:12.349 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:12.349 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:12.349 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:12.349 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:12.349 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:12.350 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:12.350 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.364+0000 7fb2ac415640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:12.350 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.364+0000 7fb2ac415640 4 set_mon_vals no callback set 2024-03-29T11:12:12.350 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.365+0000 7fb2ac415640 10 monclient: shutdown 2024-03-29T11:12:12.350 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.365+0000 7fb2ac415640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:12.350 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.365+0000 7fb2ac415640 1 -- 172.21.15.170:0/1999303920 >> v1:172.21.15.184:6789/0 conn(0x7fb2a4071de0 legacy=0x7fb2a40aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:12.350 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.365+0000 7fb2ac415640 1 -- 172.21.15.170:0/1999303920 shutdown_connections 2024-03-29T11:12:12.350 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.365+0000 7fb2ac415640 1 -- 172.21.15.170:0/1999303920 >> 172.21.15.170:0/1999303920 conn(0x7fb2a406b660 msgr2=0x7fb2a406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:12.350 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.365+0000 7fb2ac415640 1 -- 172.21.15.170:0/1999303920 shutdown_connections 2024-03-29T11:12:12.350 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.365+0000 7fb2ac415640 1 -- 172.21.15.170:0/1999303920 wait complete. 2024-03-29T11:12:12.354 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.366+0000 7fb2ac415640 10 monclient: build_initial_monmap 2024-03-29T11:12:12.354 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.366+0000 7fb2ac415640 10 monclient: monmap: 2024-03-29T11:12:12.354 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:12:12.354 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:12.354 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:12:12.354 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:12:12.354 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:12.354 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:12.354 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:12.354 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:12:12.355 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:12:12.355 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:12.355 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.367+0000 7fb2ac415640 1 librados: starting msgr at 2024-03-29T11:12:12.355 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.367+0000 7fb2ac415640 1 librados: starting objecter 2024-03-29T11:12:12.355 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.367+0000 7fb2ac415640 1 Processor -- start 2024-03-29T11:12:12.355 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.367+0000 7fb2ac415640 1 -- start start 2024-03-29T11:12:12.355 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.367+0000 7fb2ac415640 1 librados: setting wanted keys 2024-03-29T11:12:12.355 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.367+0000 7fb2ac415640 1 librados: calling monclient init 2024-03-29T11:12:12.355 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.367+0000 7fb2ac415640 10 monclient: init 2024-03-29T11:12:12.355 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.368+0000 7fb2ac415640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:12.355 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.368+0000 7fb2ac415640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:12:12.355 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.369+0000 7fb2ac415640 10 monclient(hunting): picked mon.noname-c con 0x7fb2a4071de0 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:12.355 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.369+0000 7fb2ac415640 10 monclient(hunting): picked mon.noname-b con 0x7fb2a4145ae0 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:12.355 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.369+0000 7fb2ac415640 10 monclient(hunting): picked mon.noname-a con 0x7fb2a413ad10 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:12.356 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.369+0000 7fb2ac415640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a413bdb0 con 0x7fb2a413ad10 2024-03-29T11:12:12.356 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.369+0000 7fb2ac415640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a413c020 con 0x7fb2a4145ae0 2024-03-29T11:12:12.356 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.369+0000 7fb2aa1b1640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fb2a4071de0 0x7fb2a4139f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:41086/0 (socket says 172.21.15.170:41086) 2024-03-29T11:12:12.356 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.369+0000 7fb2aa1b1640 1 -- 172.21.15.170:0/1885594162 learned_addr learned my addr 172.21.15.170:0/1885594162 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:12:12.356 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.369+0000 7fb2ac415640 1 -- 172.21.15.170:0/1885594162 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a413ce30 con 0x7fb2a4071de0 2024-03-29T11:12:12.356 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.369+0000 7fb2ac415640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:12.356 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.369+0000 7fb2ac415640 10 monclient(hunting): authenticate will time out at 913.681221s 2024-03-29T11:12:12.356 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.369+0000 7fb29affd640 10 client.?.objecter ms_handle_connect 0x7fb2a4071de0 2024-03-29T11:12:12.356 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.369+0000 7fb29affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:12.356 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.369+0000 7fb29affd640 10 client.?.objecter ms_handle_connect 0x7fb2a413ad10 2024-03-29T11:12:12.356 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.369+0000 7fb29affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:12.356 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.369+0000 7fb29affd640 10 client.?.objecter ms_handle_connect 0x7fb2a4145ae0 2024-03-29T11:12:12.356 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.369+0000 7fb29affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:12.356 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.370+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903167933 0 0) 0x7fb2a413ce30 con 0x7fb2a4071de0 2024-03-29T11:12:12.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.370+0000 7fb29affd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:12.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.370+0000 7fb29affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:12.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.370+0000 7fb29affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:12.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.370+0000 7fb29affd640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:12:12.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.370+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb288004eb0 con 0x7fb2a4071de0 2024-03-29T11:12:12.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.370+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712998002 0 0) 0x7fb2a413bdb0 con 0x7fb2a413ad10 2024-03-29T11:12:12.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.370+0000 7fb29affd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:12.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.370+0000 7fb29affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:12.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.370+0000 7fb29affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:12.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.370+0000 7fb29affd640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:12:12.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.370+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a413ce30 con 0x7fb2a413ad10 2024-03-29T11:12:12.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.370+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108871314 0 0) 0x7fb2a413c020 con 0x7fb2a4145ae0 2024-03-29T11:12:12.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.370+0000 7fb29affd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:12.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.370+0000 7fb29affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:12.357 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.370+0000 7fb29affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:12.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.370+0000 7fb29affd640 10 monclient(hunting): my global_id is 4231 2024-03-29T11:12:12.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.370+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a413bdb0 con 0x7fb2a4145ae0 2024-03-29T11:12:12.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.371+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1485862096 0 0) 0x7fb288004eb0 con 0x7fb2a4071de0 2024-03-29T11:12:12.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.371+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2a413c020 con 0x7fb2a4071de0 2024-03-29T11:12:12.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.371+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb2a0001bc0 con 0x7fb2a4071de0 2024-03-29T11:12:12.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.371+0000 7fb29affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:12.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.371+0000 7fb29affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:12.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.371+0000 7fb29affd640 10 monclient(hunting): dump: 2024-03-29T11:12:12.358 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:12.358 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:12.358 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:12.358 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:12.358 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:12.358 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:12.359 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:12.359 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:12.359 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:12.359 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:12.359 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.371+0000 7fb29affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:12.359 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.371+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3647218809 0 0) 0x7fb2a413ce30 con 0x7fb2a413ad10 2024-03-29T11:12:12.359 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.371+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb288004eb0 con 0x7fb2a413ad10 2024-03-29T11:12:12.359 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.371+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb29c001a90 con 0x7fb2a413ad10 2024-03-29T11:12:12.359 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.371+0000 7fb29affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:12.359 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.371+0000 7fb29affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:12.359 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.371+0000 7fb29affd640 10 monclient(hunting): dump: 2024-03-29T11:12:12.359 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:12.359 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:12.359 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:12.360 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:12.360 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:12.360 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:12.360 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:12.360 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:12.360 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:12.360 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:12.360 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.371+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 70633275 0 0) 0x7fb2a413bdb0 con 0x7fb2a4145ae0 2024-03-29T11:12:12.360 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2a413ce30 con 0x7fb2a4145ae0 2024-03-29T11:12:12.360 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb294001870 con 0x7fb2a4145ae0 2024-03-29T11:12:12.360 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb29affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:12.360 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb29affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:12.360 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb29affd640 10 monclient(hunting): dump: 2024-03-29T11:12:12.360 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:12.361 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:12.361 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:12.361 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:12.361 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:12.361 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:12.361 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:12.361 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:12.361 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:12.361 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:12.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2534876170 0 0) 0x7fb2a413c020 con 0x7fb2a4071de0 2024-03-29T11:12:12.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 >> v1:172.21.15.184:6789/0 conn(0x7fb2a4145ae0 legacy=0x7fb2a413c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:12.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 >> v1:172.21.15.170:6789/0 conn(0x7fb2a413ad10 legacy=0x7fb2a41e20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:12.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb29affd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:12.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb29affd640 1 monclient: found mon.c 2024-03-29T11:12:12.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb29affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:12.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb29affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:12.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a413a830 con 0x7fb2a4071de0 2024-03-29T11:12:12.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb29affd640 10 monclient: _finish_auth 0 2024-03-29T11:12:12.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb29affd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:12.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb29affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:12.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb2ac415640 5 monclient: authenticate success, global_id 4241 2024-03-29T11:12:12.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb2ac415640 10 monclient: _renew_subs 2024-03-29T11:12:12.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb2ac415640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:12.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb2ac415640 1 -- 172.21.15.170:0/1885594162 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a413c300 con 0x7fb2a4071de0 2024-03-29T11:12:12.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb2ac415640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:12.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb2ac415640 10 monclient: _renew_subs 2024-03-29T11:12:12.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb2ac415640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:12.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb2ac415640 1 -- 172.21.15.170:0/1885594162 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a4137d70 con 0x7fb2a4071de0 2024-03-29T11:12:12.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.372+0000 7fb2ac415640 1 librados: init done 2024-03-29T11:12:12.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.373+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2a0001d20 con 0x7fb2a4071de0 2024-03-29T11:12:12.363 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.373+0000 7fb29affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:12.363 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.373+0000 7fb2a89ae640 4 set_mon_vals no callback set 2024-03-29T11:12:12.363 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.373+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb2a0003a40 con 0x7fb2a4071de0 2024-03-29T11:12:12.363 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.373+0000 7fb29affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:12.363 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.373+0000 7fb29affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:12.363 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.373+0000 7fb29affd640 10 monclient: dump: 2024-03-29T11:12:12.363 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:12.363 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:12.363 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:12.363 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:12.363 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:12.363 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:12.363 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:12.363 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:12.364 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:12.364 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:12.364 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.374+0000 7fb2ac415640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:12.364 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.374+0000 7fb2ac415640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:12.364 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.374+0000 7fb2ac415640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:12.364 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.374+0000 7fb2ac415640 1 -- 172.21.15.170:0/1885594162 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb28c005080 con 0x7fb2a4071de0 2024-03-29T11:12:12.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.377+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fb2a0003ba0 con 0x7fb2a4071de0 2024-03-29T11:12:12.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.377+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fb2a0084ad0 con 0x7fb2a4071de0 2024-03-29T11:12:12.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.377+0000 7fb29affd640 10 client.4241.objecter ms_dispatch 0x7fb2a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:12.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.377+0000 7fb29affd640 3 client.4241.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:12.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.377+0000 7fb29affd640 3 client.4241.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:12.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.377+0000 7fb29affd640 20 client.4241.objecter dump_active .. 0 homeless 2024-03-29T11:12:12.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.378+0000 7fb2a99b0640 10 monclient: get_auth_request con 0x7fb28806e0d0 auth_method 0 2024-03-29T11:12:12.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.378+0000 7fb29affd640 10 client.4241.objecter ms_handle_connect 0x7fb28806e0d0 2024-03-29T11:12:12.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.385+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2a00503b0 con 0x7fb2a4071de0 2024-03-29T11:12:12.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.385+0000 7fb29affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:12.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.385+0000 7fb29affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:12.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.845+0000 7fb2ac415640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:12.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.845+0000 7fb2ac415640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:12.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.845+0000 7fb2ac415640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:12.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.845+0000 7fb2ac415640 1 -- 172.21.15.170:0/1885594162 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb28c005600 con 0x7fb2a4071de0 2024-03-29T11:12:12.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.847+0000 7fb29affd640 1 -- 172.21.15.170:0/1885594162 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fb2a00522e0 con 0x7fb2a4071de0 2024-03-29T11:12:12.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.847+0000 7fb29affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:12.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.847+0000 7fb29affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:12.826 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:12:12.826 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:12.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.851+0000 7fb2ac415640 10 librados: watch_flush enter 2024-03-29T11:12:12.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.851+0000 7fb2ac415640 10 librados: watch_flush exit 2024-03-29T11:12:12.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.851+0000 7fb2ac415640 20 client.4241.objecter shutdown clearing up homeless session... 2024-03-29T11:12:12.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.851+0000 7fb2ac415640 10 client.4241.objecter successfully canceled tick 2024-03-29T11:12:12.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.852+0000 7fb2ac415640 1 -- 172.21.15.170:0/1885594162 >> v1:172.21.15.170:6816/3298994444 conn(0x7fb28806e0d0 legacy=0x7fb288070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:12.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.852+0000 7fb2ac415640 10 monclient: shutdown 2024-03-29T11:12:12.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.852+0000 7fb2ac415640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:12.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.852+0000 7fb2ac415640 1 -- 172.21.15.170:0/1885594162 >> v1:172.21.15.170:6790/0 conn(0x7fb2a4071de0 legacy=0x7fb2a4139f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:12.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.852+0000 7fb2ac415640 1 -- 172.21.15.170:0/1885594162 shutdown_connections 2024-03-29T11:12:12.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.852+0000 7fb2ac415640 1 -- 172.21.15.170:0/1885594162 >> 172.21.15.170:0/1885594162 conn(0x7fb2a406b660 msgr2=0x7fb2a413f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:12.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.852+0000 7fb2ac415640 1 -- 172.21.15.170:0/1885594162 shutdown_connections 2024-03-29T11:12:12.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.852+0000 7fb2ac415640 1 -- 172.21.15.170:0/1885594162 wait complete. 2024-03-29T11:12:12.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:12.852+0000 7fb2ac415640 1 librados: shutdown 2024-03-29T11:12:12.848 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:19.657 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:19.806 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.827+0000 7efea0c06640 10 monclient: get_monmap_and_config 2024-03-29T11:12:19.806 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.827+0000 7efea0c06640 10 monclient: build_initial_monmap 2024-03-29T11:12:19.806 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.827+0000 7efea0c06640 10 monclient: monmap: 2024-03-29T11:12:19.807 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:12:19.807 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:19.807 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:12:19.829175+0000 2024-03-29T11:12:19.807 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:12:19.829175+0000 2024-03-29T11:12:19.807 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:19.807 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:19.807 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:19.807 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:12:19.807 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:12:19.807 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:19.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.828+0000 7efea0c06640 1 Processor -- start 2024-03-29T11:12:19.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.828+0000 7efea0c06640 1 -- start start 2024-03-29T11:12:19.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.828+0000 7efea0c06640 10 monclient: init 2024-03-29T11:12:19.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.829+0000 7efea0c06640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:19.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.829+0000 7efea0c06640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:12:19.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.829+0000 7efe9a59c640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7efe9c0b0500 0x7efe9c072050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:39376/0 (socket says 172.21.15.170:39376) 2024-03-29T11:12:19.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.829+0000 7efe9a59c640 1 -- 172.21.15.170:0/1621319482 learned_addr learned my addr 172.21.15.170:0/1621319482 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:12:19.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.829+0000 7efea0c06640 10 monclient(hunting): picked mon.noname-b con 0x7efe9c0b0500 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:19.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.830+0000 7efea0c06640 10 monclient(hunting): picked mon.noname-a con 0x7efe9c0b08e0 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:19.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.830+0000 7efea0c06640 10 monclient(hunting): picked mon.noname-c con 0x7efe9c0aec70 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:19.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.830+0000 7efea0c06640 1 -- 172.21.15.170:0/1621319482 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c0ae7e0 con 0x7efe9c0b08e0 2024-03-29T11:12:19.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.830+0000 7efea0c06640 1 -- 172.21.15.170:0/1621319482 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c13daf0 con 0x7efe9c0aec70 2024-03-29T11:12:19.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.830+0000 7efea0c06640 1 -- 172.21.15.170:0/1621319482 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c0accf0 con 0x7efe9c0b0500 2024-03-29T11:12:19.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.830+0000 7efea0c06640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:19.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.830+0000 7efea0c06640 10 monclient(hunting): authenticate will time out at 921.142667s 2024-03-29T11:12:19.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.830+0000 7efe99d9b640 1 -- 172.21.15.170:0/1621319482 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020349852 0 0) 0x7efe9c0accf0 con 0x7efe9c0b0500 2024-03-29T11:12:19.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.830+0000 7efe99d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:12:19.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.830+0000 7efe99d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:19.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.830+0000 7efe99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:19.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.830+0000 7efe99d9b640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:12:19.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.830+0000 7efe99d9b640 1 -- 172.21.15.170:0/1621319482 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe88003520 con 0x7efe9c0b0500 2024-03-29T11:12:19.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.831+0000 7efe99d9b640 1 -- 172.21.15.170:0/1621319482 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3519712524 0 0) 0x7efe88003520 con 0x7efe9c0b0500 2024-03-29T11:12:19.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.831+0000 7efe99d9b640 1 -- 172.21.15.170:0/1621319482 >> v1:172.21.15.184:6789/0 conn(0x7efe9c0aec70 legacy=0x7efe9c13ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:19.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.831+0000 7efe99d9b640 1 -- 172.21.15.170:0/1621319482 >> v1:172.21.15.170:6789/0 conn(0x7efe9c0b08e0 legacy=0x7efe9c13d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:12:19.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.831+0000 7efe99d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:12:19.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.831+0000 7efe99d9b640 1 monclient: found mon.noname-b 2024-03-29T11:12:19.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.831+0000 7efe99d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:19.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.831+0000 7efe99d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:12:19.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.831+0000 7efe99d9b640 1 -- 172.21.15.170:0/1621319482 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c140620 con 0x7efe9c0b0500 2024-03-29T11:12:19.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.831+0000 7efe99d9b640 10 monclient: _finish_auth 0 2024-03-29T11:12:19.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.831+0000 7efe99d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:12:19.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.831+0000 7efe99d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:19.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.831+0000 7efe99d9b640 1 -- 172.21.15.170:0/1621319482 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7efe8c001a10 con 0x7efe9c0b0500 2024-03-29T11:12:19.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.831+0000 7efe99d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:19.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.831+0000 7efe99d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:19.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.831+0000 7efe99d9b640 10 monclient: dump: 2024-03-29T11:12:19.813 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:19.813 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:19.813 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:19.813 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:19.813 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:19.813 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:19.813 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:19.813 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:19.813 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:19.813 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:19.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.831+0000 7efea0c06640 5 monclient: authenticate success, global_id 4244 2024-03-29T11:12:19.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.831+0000 7efea0c06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:19.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.832+0000 7efe99d9b640 1 -- 172.21.15.170:0/1621319482 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe8c001ca0 con 0x7efe9c0b0500 2024-03-29T11:12:19.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.832+0000 7efe99d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:19.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.832+0000 7efe99d9b640 1 -- 172.21.15.170:0/1621319482 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7efe8c002030 con 0x7efe9c0b0500 2024-03-29T11:12:19.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.832+0000 7efea0c06640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:19.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.832+0000 7efea0c06640 4 set_mon_vals no callback set 2024-03-29T11:12:19.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.832+0000 7efe99d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:19.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.832+0000 7efe99d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:19.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.832+0000 7efe99d9b640 10 monclient: dump: 2024-03-29T11:12:19.814 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:19.814 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:19.814 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:19.814 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:19.815 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:19.815 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:19.815 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:19.815 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:19.815 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:19.815 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:19.822 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.832+0000 7efea0c06640 10 monclient: shutdown 2024-03-29T11:12:19.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.832+0000 7efea0c06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:19.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.832+0000 7efea0c06640 1 -- 172.21.15.170:0/1621319482 >> v1:172.21.15.170:6790/0 conn(0x7efe9c0b0500 legacy=0x7efe9c072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:19.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.833+0000 7efea0c06640 1 -- 172.21.15.170:0/1621319482 shutdown_connections 2024-03-29T11:12:19.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.833+0000 7efea0c06640 1 -- 172.21.15.170:0/1621319482 >> 172.21.15.170:0/1621319482 conn(0x7efe9c06b660 msgr2=0x7efe9c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:19.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.834+0000 7efea0c06640 1 -- 172.21.15.170:0/1621319482 shutdown_connections 2024-03-29T11:12:19.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.834+0000 7efea0c06640 1 -- 172.21.15.170:0/1621319482 wait complete. 2024-03-29T11:12:19.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.835+0000 7efea0c06640 10 monclient: build_initial_monmap 2024-03-29T11:12:19.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.835+0000 7efea0c06640 10 monclient: monmap: 2024-03-29T11:12:19.824 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:12:19.824 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:19.824 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:12:19.824 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:12:19.824 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:19.824 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:19.824 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:19.824 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:12:19.824 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:12:19.824 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:19.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.836+0000 7efea0c06640 1 librados: starting msgr at 2024-03-29T11:12:19.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.836+0000 7efea0c06640 1 librados: starting objecter 2024-03-29T11:12:19.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.836+0000 7efea0c06640 1 Processor -- start 2024-03-29T11:12:19.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.836+0000 7efea0c06640 1 -- start start 2024-03-29T11:12:19.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.836+0000 7efea0c06640 1 librados: setting wanted keys 2024-03-29T11:12:19.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.836+0000 7efea0c06640 1 librados: calling monclient init 2024-03-29T11:12:19.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.836+0000 7efea0c06640 10 monclient: init 2024-03-29T11:12:19.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.837+0000 7efea0c06640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:19.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.837+0000 7efea0c06640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:12:19.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.837+0000 7efea0c06640 10 monclient(hunting): picked mon.noname-b con 0x7efe9c0aec70 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:19.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.837+0000 7efea0c06640 10 monclient(hunting): picked mon.noname-c con 0x7efe9c0b0500 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:19.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.837+0000 7efea0c06640 10 monclient(hunting): picked mon.noname-a con 0x7efe9c13a470 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:19.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.837+0000 7efe9a59c640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7efe9c0aec70 0x7efe9c11b1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:33298/0 (socket says 172.21.15.170:33298) 2024-03-29T11:12:19.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.837+0000 7efe9a59c640 1 -- 172.21.15.170:0/410310684 learned_addr learned my addr 172.21.15.170:0/410310684 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:12:19.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.837+0000 7efe91d9b640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7efe9c0b0500 0x7efe9c139d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:40204/0 (socket says 172.21.15.170:40204) 2024-03-29T11:12:19.826 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.837+0000 7efea0c06640 1 -- 172.21.15.170:0/410310684 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c11b8d0 con 0x7efe9c13a470 2024-03-29T11:12:19.826 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.837+0000 7efea0c06640 1 -- 172.21.15.170:0/410310684 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c1d3930 con 0x7efe9c0aec70 2024-03-29T11:12:19.826 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.838+0000 7efe937fe640 10 client.?.objecter ms_handle_connect 0x7efe9c0aec70 2024-03-29T11:12:19.826 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.838+0000 7efe937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:19.826 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.838+0000 7efea0c06640 1 -- 172.21.15.170:0/410310684 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c13a9e0 con 0x7efe9c0b0500 2024-03-29T11:12:19.826 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.838+0000 7efea0c06640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:19.826 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.838+0000 7efea0c06640 10 monclient(hunting): authenticate will time out at 921.150232s 2024-03-29T11:12:19.826 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.838+0000 7efe937fe640 10 client.?.objecter ms_handle_connect 0x7efe9c0b0500 2024-03-29T11:12:19.826 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.838+0000 7efe937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:19.826 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.838+0000 7efe937fe640 1 -- 172.21.15.170:0/410310684 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604775194 0 0) 0x7efe9c13a9e0 con 0x7efe9c0b0500 2024-03-29T11:12:19.826 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.838+0000 7efe937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:19.826 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.838+0000 7efe937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:19.826 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.838+0000 7efe937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:19.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.838+0000 7efe937fe640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:12:19.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.838+0000 7efe937fe640 1 -- 172.21.15.170:0/410310684 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe7c004eb0 con 0x7efe9c0b0500 2024-03-29T11:12:19.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 1 -- 172.21.15.170:0/410310684 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2031629241 0 0) 0x7efe7c004eb0 con 0x7efe9c0b0500 2024-03-29T11:12:19.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 1 -- 172.21.15.170:0/410310684 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe9c13a9e0 con 0x7efe9c0b0500 2024-03-29T11:12:19.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 1 -- 172.21.15.170:0/410310684 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7efe840017d0 con 0x7efe9c0b0500 2024-03-29T11:12:19.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:19.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:19.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 10 monclient(hunting): dump: 2024-03-29T11:12:19.827 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:19.827 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:19.827 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:19.827 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:19.827 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:19.827 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:19.827 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:19.828 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:19.828 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:19.828 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:19.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:19.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 1 -- 172.21.15.170:0/410310684 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3397446898 0 0) 0x7efe9c13a9e0 con 0x7efe9c0b0500 2024-03-29T11:12:19.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 1 -- 172.21.15.170:0/410310684 >> v1:172.21.15.184:6789/0 conn(0x7efe9c0aec70 legacy=0x7efe9c11b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:19.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 1 -- 172.21.15.170:0/410310684 >> v1:172.21.15.170:6789/0 conn(0x7efe9c13a470 legacy=0x7efe9c1deb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:19.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 10 monclient: _finish_hunting 0 2024-03-29T11:12:19.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 1 monclient: found mon.c 2024-03-29T11:12:19.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:19.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:19.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 1 -- 172.21.15.170:0/410310684 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c13ab70 con 0x7efe9c0b0500 2024-03-29T11:12:19.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 10 monclient: _finish_auth 0 2024-03-29T11:12:19.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 10 monclient: _check_auth_tickets 2024-03-29T11:12:19.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:19.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 10 client.?.objecter ms_handle_connect 0x7efe9c13a470 2024-03-29T11:12:19.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.839+0000 7efe937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:19.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.840+0000 7efea0c06640 5 monclient: authenticate success, global_id 4247 2024-03-29T11:12:19.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.840+0000 7efea0c06640 10 monclient: _renew_subs 2024-03-29T11:12:19.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.840+0000 7efea0c06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:19.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.840+0000 7efea0c06640 1 -- 172.21.15.170:0/410310684 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe9c13ad60 con 0x7efe9c0b0500 2024-03-29T11:12:19.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.840+0000 7efea0c06640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:19.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.840+0000 7efea0c06640 10 monclient: _renew_subs 2024-03-29T11:12:19.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.840+0000 7efea0c06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:19.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.840+0000 7efea0c06640 1 -- 172.21.15.170:0/410310684 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9c070590 con 0x7efe9c0b0500 2024-03-29T11:12:19.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.840+0000 7efea0c06640 1 librados: init done 2024-03-29T11:12:19.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.840+0000 7efe937fe640 1 -- 172.21.15.170:0/410310684 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe84001d60 con 0x7efe9c0b0500 2024-03-29T11:12:19.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.840+0000 7efe937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:19.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.840+0000 7efe9959a640 4 set_mon_vals no callback set 2024-03-29T11:12:19.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.840+0000 7efe937fe640 1 -- 172.21.15.170:0/410310684 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7efe84004940 con 0x7efe9c0b0500 2024-03-29T11:12:19.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.840+0000 7efe937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:19.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.840+0000 7efe937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:19.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.840+0000 7efe937fe640 10 monclient: dump: 2024-03-29T11:12:19.830 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:19.830 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:19.830 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:19.830 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:19.830 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:19.830 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:19.830 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:19.830 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:19.830 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:19.830 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:19.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.841+0000 7efea0c06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:19.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.841+0000 7efea0c06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:19.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.841+0000 7efea0c06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:19.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.841+0000 7efea0c06640 1 -- 172.21.15.170:0/410310684 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe80005080 con 0x7efe9c0b0500 2024-03-29T11:12:19.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.843+0000 7efe937fe640 1 -- 172.21.15.170:0/410310684 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7efe8401a990 con 0x7efe9c0b0500 2024-03-29T11:12:19.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.844+0000 7efe9a59c640 10 monclient: get_auth_request con 0x7efe7c06dbf0 auth_method 0 2024-03-29T11:12:19.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.844+0000 7efe937fe640 1 -- 172.21.15.170:0/410310684 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7efe84084980 con 0x7efe9c0b0500 2024-03-29T11:12:19.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.844+0000 7efe937fe640 10 client.4247.objecter ms_dispatch 0x7efe9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:19.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.844+0000 7efe937fe640 3 client.4247.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:19.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.844+0000 7efe937fe640 3 client.4247.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:19.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.844+0000 7efe937fe640 20 client.4247.objecter dump_active .. 0 homeless 2024-03-29T11:12:19.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.844+0000 7efe937fe640 10 client.4247.objecter ms_handle_connect 0x7efe7c06dbf0 2024-03-29T11:12:19.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.850+0000 7efe937fe640 1 -- 172.21.15.170:0/410310684 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe84050090 con 0x7efe9c0b0500 2024-03-29T11:12:19.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.850+0000 7efe937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:19.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:19.850+0000 7efe937fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:20.140 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.162+0000 7efea0c06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:20.140 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.162+0000 7efea0c06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:20.140 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.162+0000 7efea0c06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:20.140 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.162+0000 7efea0c06640 1 -- 172.21.15.170:0/410310684 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe80005330 con 0x7efe9c0b0500 2024-03-29T11:12:20.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.162+0000 7efe937fe640 1 -- 172.21.15.170:0/410310684 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7efe84052190 con 0x7efe9c0b0500 2024-03-29T11:12:20.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.162+0000 7efe937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:20.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.162+0000 7efe937fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:20.141 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:12:20.141 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:20.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.165+0000 7efea0c06640 10 librados: watch_flush enter 2024-03-29T11:12:20.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.165+0000 7efea0c06640 10 librados: watch_flush exit 2024-03-29T11:12:20.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.165+0000 7efea0c06640 20 client.4247.objecter shutdown clearing up homeless session... 2024-03-29T11:12:20.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.165+0000 7efea0c06640 10 client.4247.objecter successfully canceled tick 2024-03-29T11:12:20.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.165+0000 7efea0c06640 1 -- 172.21.15.170:0/410310684 >> v1:172.21.15.170:6816/3298994444 conn(0x7efe7c06dbf0 legacy=0x7efe7c0700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:20.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.165+0000 7efea0c06640 10 monclient: shutdown 2024-03-29T11:12:20.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.165+0000 7efea0c06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:20.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.165+0000 7efea0c06640 1 -- 172.21.15.170:0/410310684 >> v1:172.21.15.170:6790/0 conn(0x7efe9c0b0500 legacy=0x7efe9c139d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:20.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.165+0000 7efea0c06640 1 -- 172.21.15.170:0/410310684 shutdown_connections 2024-03-29T11:12:20.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.165+0000 7efea0c06640 1 -- 172.21.15.170:0/410310684 >> 172.21.15.170:0/410310684 conn(0x7efe9c06b660 msgr2=0x7efe9c13de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:20.145 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.166+0000 7efea0c06640 1 -- 172.21.15.170:0/410310684 shutdown_connections 2024-03-29T11:12:20.145 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.166+0000 7efea0c06640 1 -- 172.21.15.170:0/410310684 wait complete. 2024-03-29T11:12:20.145 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:20.166+0000 7efea0c06640 1 librados: shutdown 2024-03-29T11:12:20.160 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:26.968 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:27.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.140+0000 7faf4b59e640 10 monclient: get_monmap_and_config 2024-03-29T11:12:27.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.140+0000 7faf4b59e640 10 monclient: build_initial_monmap 2024-03-29T11:12:27.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.140+0000 7faf4b59e640 10 monclient: monmap: 2024-03-29T11:12:27.119 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:12:27.119 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:27.119 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:12:27.142083+0000 2024-03-29T11:12:27.119 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:12:27.142083+0000 2024-03-29T11:12:27.119 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:27.119 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:27.119 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:27.119 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:12:27.119 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:12:27.119 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:27.120 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.141+0000 7faf4b59e640 1 Processor -- start 2024-03-29T11:12:27.120 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.141+0000 7faf4b59e640 1 -- start start 2024-03-29T11:12:27.120 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.141+0000 7faf4b59e640 10 monclient: init 2024-03-29T11:12:27.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.142+0000 7faf4b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:27.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.142+0000 7faf4b59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:12:27.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.143+0000 7faf4b59e640 10 monclient(hunting): picked mon.noname-b con 0x7faf4c0b4920 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:27.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.143+0000 7faf4b59e640 10 monclient(hunting): picked mon.noname-a con 0x7faf4c0b4200 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:27.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.143+0000 7faf4b59e640 10 monclient(hunting): picked mon.noname-c con 0x7faf4c13f9a0 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:27.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.143+0000 7faf4b59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4c0b0930 con 0x7faf4c0b4200 2024-03-29T11:12:27.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.143+0000 7faf4b59e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4c0af460 con 0x7faf4c13f9a0 2024-03-29T11:12:27.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.143+0000 7faf4b59e640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4c0aeda0 con 0x7faf4c0b4920 2024-03-29T11:12:27.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.143+0000 7faf4b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:27.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.143+0000 7faf4b59e640 10 monclient(hunting): authenticate will time out at 928.455485s 2024-03-29T11:12:27.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.144+0000 7faf4ad9d640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7faf4c13f9a0 0x7faf4c14d8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:38652/0 (socket says 172.21.15.170:38652) 2024-03-29T11:12:27.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.144+0000 7faf4ad9d640 1 -- 172.21.15.170:0/752619239 learned_addr learned my addr 172.21.15.170:0/752619239 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:12:27.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 1 -- 172.21.15.170:0/752619239 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703499303 0 0) 0x7faf4c0af460 con 0x7faf4c13f9a0 2024-03-29T11:12:27.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:27.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:27.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:27.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:12:27.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 1 -- 172.21.15.170:0/752619239 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf34003520 con 0x7faf4c13f9a0 2024-03-29T11:12:27.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 1 -- 172.21.15.170:0/752619239 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809659718 0 0) 0x7faf4c0b0930 con 0x7faf4c0b4200 2024-03-29T11:12:27.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:27.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:27.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:27.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:12:27.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 1 -- 172.21.15.170:0/752619239 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf4c0af460 con 0x7faf4c0b4200 2024-03-29T11:12:27.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 1 -- 172.21.15.170:0/752619239 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991526928 0 0) 0x7faf4c0aeda0 con 0x7faf4c0b4920 2024-03-29T11:12:27.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:27.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:27.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:27.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:12:27.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 1 -- 172.21.15.170:0/752619239 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf4c0b0930 con 0x7faf4c0b4920 2024-03-29T11:12:27.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 1 -- 172.21.15.170:0/752619239 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1220015020 0 0) 0x7faf4c0af460 con 0x7faf4c0b4200 2024-03-29T11:12:27.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 1 -- 172.21.15.170:0/752619239 >> v1:172.21.15.170:6790/0 conn(0x7faf4c0b4920 legacy=0x7faf4c13f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.147+0000 7faf4959a640 1 -- 172.21.15.170:0/752619239 >> v1:172.21.15.184:6789/0 conn(0x7faf4c13f9a0 legacy=0x7faf4c14d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4959a640 10 monclient: _finish_hunting 0 2024-03-29T11:12:27.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4959a640 1 monclient: found mon.noname-a 2024-03-29T11:12:27.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:27.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:12:27.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4959a640 1 -- 172.21.15.170:0/752619239 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4c0af980 con 0x7faf4c0b4200 2024-03-29T11:12:27.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4959a640 10 monclient: _finish_auth 0 2024-03-29T11:12:27.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4959a640 10 monclient: _check_auth_tickets 2024-03-29T11:12:27.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:27.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4959a640 1 -- 172.21.15.170:0/752619239 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7faf400015f0 con 0x7faf4c0b4200 2024-03-29T11:12:27.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4b59e640 5 monclient: authenticate success, global_id 4245 2024-03-29T11:12:27.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:27.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:27.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:27.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4959a640 10 monclient: dump: 2024-03-29T11:12:27.135 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:27.135 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:27.135 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:27.135 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:27.135 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:27.135 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:27.135 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:27.135 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:27.135 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:27.135 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:27.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4959a640 1 -- 172.21.15.170:0/752619239 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf40001810 con 0x7faf4c0b4200 2024-03-29T11:12:27.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:27.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:27.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4959a640 1 -- 172.21.15.170:0/752619239 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7faf40001ba0 con 0x7faf4c0b4200 2024-03-29T11:12:27.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4b59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:27.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.148+0000 7faf4b59e640 4 set_mon_vals no callback set 2024-03-29T11:12:27.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.149+0000 7faf4b59e640 10 monclient: shutdown 2024-03-29T11:12:27.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.149+0000 7faf4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:27.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.149+0000 7faf4959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:27.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.149+0000 7faf4959a640 10 monclient: dump: 2024-03-29T11:12:27.136 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:27.136 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:27.136 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:27.136 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:27.136 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:27.137 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:27.137 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:27.137 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:27.137 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:27.137 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:27.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.149+0000 7faf4b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:27.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.149+0000 7faf4b59e640 1 -- 172.21.15.170:0/752619239 >> v1:172.21.15.170:6789/0 conn(0x7faf4c0b4200 legacy=0x7faf4c0b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.151+0000 7faf4b59e640 1 -- 172.21.15.170:0/752619239 shutdown_connections 2024-03-29T11:12:27.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.151+0000 7faf4b59e640 1 -- 172.21.15.170:0/752619239 >> 172.21.15.170:0/752619239 conn(0x7faf4c06d230 msgr2=0x7faf4c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:27.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.151+0000 7faf4b59e640 1 -- 172.21.15.170:0/752619239 shutdown_connections 2024-03-29T11:12:27.138 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.151+0000 7faf4b59e640 1 -- 172.21.15.170:0/752619239 wait complete. 2024-03-29T11:12:27.138 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.153+0000 7faf4b59e640 10 monclient: build_initial_monmap 2024-03-29T11:12:27.138 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.153+0000 7faf4b59e640 10 monclient: monmap: 2024-03-29T11:12:27.138 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:12:27.138 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:27.138 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:12:27.138 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:12:27.138 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:27.139 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:27.139 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:27.139 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:12:27.139 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:12:27.139 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:27.146 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.153+0000 7faf4b59e640 1 librados: starting msgr at 2024-03-29T11:12:27.146 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.153+0000 7faf4b59e640 1 librados: starting objecter 2024-03-29T11:12:27.146 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.153+0000 7faf4b59e640 1 Processor -- start 2024-03-29T11:12:27.146 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.155+0000 7faf4b59e640 1 -- start start 2024-03-29T11:12:27.146 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.155+0000 7faf4b59e640 1 librados: setting wanted keys 2024-03-29T11:12:27.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.155+0000 7faf4b59e640 1 librados: calling monclient init 2024-03-29T11:12:27.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.155+0000 7faf4b59e640 10 monclient: init 2024-03-29T11:12:27.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.156+0000 7faf4b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:27.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.156+0000 7faf4b59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:27.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.156+0000 7faf4b59e640 10 monclient(hunting): picked mon.noname-b con 0x7faf4c0b4200 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:27.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.156+0000 7faf4b59e640 10 monclient(hunting): picked mon.noname-a con 0x7faf4c0b4920 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:27.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.160+0000 7faf4b59e640 10 monclient(hunting): picked mon.noname-c con 0x7faf4c13f9a0 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:27.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.160+0000 7faf4b59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4c145ba0 con 0x7faf4c0b4920 2024-03-29T11:12:27.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.160+0000 7faf4ad9d640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7faf4c13f9a0 0x7faf4c0ae1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:37179/0 (socket says 172.21.15.170:37179) 2024-03-29T11:12:27.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.160+0000 7faf4ad9d640 1 -- 172.21.15.170:0/2267546179 learned_addr learned my addr 172.21.15.170:0/2267546179 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:12:27.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.160+0000 7faf3affd640 10 client.?.objecter ms_handle_connect 0x7faf4c13f9a0 2024-03-29T11:12:27.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.160+0000 7faf3affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:27.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.160+0000 7faf4a59c640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7faf4c0b4200 0x7faf4c1d7030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:38846/0 (socket says 172.21.15.170:38846) 2024-03-29T11:12:27.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.160+0000 7faf4b59e640 1 -- 172.21.15.170:0/2267546179 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4c0ae8e0 con 0x7faf4c0b4200 2024-03-29T11:12:27.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.160+0000 7faf4b59e640 1 -- 172.21.15.170:0/2267546179 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4c1460c0 con 0x7faf4c13f9a0 2024-03-29T11:12:27.149 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.160+0000 7faf3affd640 10 client.?.objecter ms_handle_connect 0x7faf4c0b4200 2024-03-29T11:12:27.149 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.160+0000 7faf3affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:27.149 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.160+0000 7faf4b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:27.149 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.160+0000 7faf4b59e640 10 monclient(hunting): authenticate will time out at 928.472810s 2024-03-29T11:12:27.149 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 1 -- 172.21.15.170:0/2267546179 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547529399 0 0) 0x7faf4c1460c0 con 0x7faf4c13f9a0 2024-03-29T11:12:27.149 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:27.149 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:27.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:27.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:12:27.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 1 -- 172.21.15.170:0/2267546179 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf2c004eb0 con 0x7faf4c13f9a0 2024-03-29T11:12:27.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 10 client.?.objecter ms_handle_connect 0x7faf4c0b4920 2024-03-29T11:12:27.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:27.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 1 -- 172.21.15.170:0/2267546179 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 950393946 0 0) 0x7faf2c004eb0 con 0x7faf4c13f9a0 2024-03-29T11:12:27.151 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 1 -- 172.21.15.170:0/2267546179 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf4c1460c0 con 0x7faf4c13f9a0 2024-03-29T11:12:27.151 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 1 -- 172.21.15.170:0/2267546179 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7faf44001af0 con 0x7faf4c13f9a0 2024-03-29T11:12:27.151 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:27.151 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:27.151 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 10 monclient(hunting): dump: 2024-03-29T11:12:27.151 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:27.151 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:27.152 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:27.152 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:27.152 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:27.152 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:27.152 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:27.152 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:27.153 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:27.153 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:27.153 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:27.153 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 1 -- 172.21.15.170:0/2267546179 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345051908 0 0) 0x7faf4c145ba0 con 0x7faf4c0b4920 2024-03-29T11:12:27.153 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:27.153 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:27.153 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:27.154 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:12:27.154 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.161+0000 7faf3affd640 1 -- 172.21.15.170:0/2267546179 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf2c004eb0 con 0x7faf4c0b4920 2024-03-29T11:12:27.154 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.162+0000 7faf3affd640 1 -- 172.21.15.170:0/2267546179 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 243747475 0 0) 0x7faf4c1460c0 con 0x7faf4c13f9a0 2024-03-29T11:12:27.154 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.162+0000 7faf3affd640 1 -- 172.21.15.170:0/2267546179 >> v1:172.21.15.184:6789/0 conn(0x7faf4c0b4200 legacy=0x7faf4c1d7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.154 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.162+0000 7faf3affd640 1 -- 172.21.15.170:0/2267546179 >> v1:172.21.15.170:6789/0 conn(0x7faf4c0b4920 legacy=0x7faf4c0adac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.154 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.162+0000 7faf3affd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:27.155 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.162+0000 7faf3affd640 1 monclient: found mon.c 2024-03-29T11:12:27.155 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.162+0000 7faf3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:27.155 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.162+0000 7faf3affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:27.155 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.162+0000 7faf3affd640 1 -- 172.21.15.170:0/2267546179 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4c1457c0 con 0x7faf4c13f9a0 2024-03-29T11:12:27.155 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.162+0000 7faf3affd640 10 monclient: _finish_auth 0 2024-03-29T11:12:27.155 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.162+0000 7faf3affd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:27.156 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.162+0000 7faf3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:27.156 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.162+0000 7faf4b59e640 5 monclient: authenticate success, global_id 4253 2024-03-29T11:12:27.156 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.162+0000 7faf4b59e640 10 monclient: _renew_subs 2024-03-29T11:12:27.156 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.162+0000 7faf4b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:27.156 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.162+0000 7faf4b59e640 1 -- 172.21.15.170:0/2267546179 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf4c1d7740 con 0x7faf4c13f9a0 2024-03-29T11:12:27.156 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.162+0000 7faf4b59e640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:27.156 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.163+0000 7faf4b59e640 10 monclient: _renew_subs 2024-03-29T11:12:27.157 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.163+0000 7faf4b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:27.157 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.163+0000 7faf4b59e640 1 -- 172.21.15.170:0/2267546179 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf4c145450 con 0x7faf4c13f9a0 2024-03-29T11:12:27.157 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.163+0000 7faf4b59e640 1 librados: init done 2024-03-29T11:12:27.157 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.164+0000 7faf4b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:27.157 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.164+0000 7faf4b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:27.157 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.164+0000 7faf4b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:27.158 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.164+0000 7faf4b59e640 1 -- 172.21.15.170:0/2267546179 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf4c145a10 con 0x7faf4c13f9a0 2024-03-29T11:12:27.158 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.164+0000 7faf3affd640 1 -- 172.21.15.170:0/2267546179 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf44002060 con 0x7faf4c13f9a0 2024-03-29T11:12:27.158 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.164+0000 7faf3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:27.158 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.164+0000 7faf3affd640 1 -- 172.21.15.170:0/2267546179 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7faf44004d00 con 0x7faf4c13f9a0 2024-03-29T11:12:27.158 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.164+0000 7faf3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:27.158 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.164+0000 7faf3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:27.159 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.164+0000 7faf3affd640 10 monclient: dump: 2024-03-29T11:12:27.159 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:27.159 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:27.159 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:27.159 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:27.159 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:27.159 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:27.160 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:27.160 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:27.160 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:27.161 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:27.161 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.165+0000 7faf48d99640 4 set_mon_vals no callback set 2024-03-29T11:12:27.161 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.165+0000 7faf3affd640 1 -- 172.21.15.170:0/2267546179 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7faf44004f20 con 0x7faf4c13f9a0 2024-03-29T11:12:27.161 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.166+0000 7faf3affd640 1 -- 172.21.15.170:0/2267546179 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7faf44054b70 con 0x7faf4c13f9a0 2024-03-29T11:12:27.161 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.166+0000 7faf3affd640 10 client.4253.objecter ms_dispatch 0x7faf4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:27.161 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.166+0000 7faf3affd640 3 client.4253.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:27.161 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.166+0000 7faf3affd640 3 client.4253.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:27.162 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.166+0000 7faf3affd640 20 client.4253.objecter dump_active .. 0 homeless 2024-03-29T11:12:27.162 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.166+0000 7faf4a59c640 10 monclient: get_auth_request con 0x7faf2c06dd40 auth_method 0 2024-03-29T11:12:27.162 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.167+0000 7faf3affd640 10 client.4253.objecter ms_handle_connect 0x7faf2c06dd40 2024-03-29T11:12:27.163 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.175+0000 7faf3affd640 1 -- 172.21.15.170:0/2267546179 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf44050e10 con 0x7faf4c13f9a0 2024-03-29T11:12:27.163 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.175+0000 7faf3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:27.163 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.175+0000 7faf3affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:27.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.503+0000 7faf4b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:27.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.503+0000 7faf4b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:27.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.503+0000 7faf4b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:12:27.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.503+0000 7faf4b59e640 1 -- 172.21.15.170:0/2267546179 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf4c145d50 con 0x7faf4c13f9a0 2024-03-29T11:12:27.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.504+0000 7faf3affd640 1 -- 172.21.15.170:0/2267546179 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7faf4c145d50 con 0x7faf4c13f9a0 2024-03-29T11:12:27.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.504+0000 7faf3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:27.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.504+0000 7faf3affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:27.483 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:12:27.483 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:27.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.508+0000 7faf4b59e640 10 librados: watch_flush enter 2024-03-29T11:12:27.489 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.510+0000 7faf4b59e640 10 librados: watch_flush exit 2024-03-29T11:12:27.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.510+0000 7faf4b59e640 20 client.4253.objecter shutdown clearing up homeless session... 2024-03-29T11:12:27.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.510+0000 7faf4b59e640 10 client.4253.objecter successfully canceled tick 2024-03-29T11:12:27.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.511+0000 7faf4b59e640 1 -- 172.21.15.170:0/2267546179 >> v1:172.21.15.170:6816/3298994444 conn(0x7faf2c06dd40 legacy=0x7faf2c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.511+0000 7faf4b59e640 10 monclient: shutdown 2024-03-29T11:12:27.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.511+0000 7faf4b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:27.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.511+0000 7faf4b59e640 1 -- 172.21.15.170:0/2267546179 >> v1:172.21.15.170:6790/0 conn(0x7faf4c13f9a0 legacy=0x7faf4c0ae1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.511+0000 7faf4b59e640 1 -- 172.21.15.170:0/2267546179 shutdown_connections 2024-03-29T11:12:27.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.511+0000 7faf4b59e640 1 -- 172.21.15.170:0/2267546179 >> 172.21.15.170:0/2267546179 conn(0x7faf4c06d230 msgr2=0x7faf4c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:27.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.511+0000 7faf4b59e640 1 -- 172.21.15.170:0/2267546179 shutdown_connections 2024-03-29T11:12:27.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.511+0000 7faf4b59e640 1 -- 172.21.15.170:0/2267546179 wait complete. 2024-03-29T11:12:27.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:27.511+0000 7faf4b59e640 1 librados: shutdown 2024-03-29T11:12:27.508 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:34.316 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:34.467 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.490+0000 7f12cb59e640 10 monclient: get_monmap_and_config 2024-03-29T11:12:34.467 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.490+0000 7f12cb59e640 10 monclient: build_initial_monmap 2024-03-29T11:12:34.468 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.490+0000 7f12cb59e640 10 monclient: monmap: 2024-03-29T11:12:34.468 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:12:34.468 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:34.468 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:12:34.490676+0000 2024-03-29T11:12:34.468 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:12:34.490676+0000 2024-03-29T11:12:34.468 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:34.468 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:34.468 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:34.468 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:12:34.468 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:12:34.468 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:34.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.491+0000 7f12cb59e640 1 Processor -- start 2024-03-29T11:12:34.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.491+0000 7f12cb59e640 1 -- start start 2024-03-29T11:12:34.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.491+0000 7f12cb59e640 10 monclient: init 2024-03-29T11:12:34.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.491+0000 7f12cb59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:34.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.491+0000 7f12cb59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:34.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.491+0000 7f12cb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f12cc0a9750 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:34.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.491+0000 7f12cb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f12cc0add40 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:34.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.491+0000 7f12cb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f12cc0aad20 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:34.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.491+0000 7f12cb59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12cc149870 con 0x7f12cc0add40 2024-03-29T11:12:34.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.491+0000 7f12cb59e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12cc0abf80 con 0x7f12cc0a9750 2024-03-29T11:12:34.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.491+0000 7f12cb59e640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12cc149ac0 con 0x7f12cc0aad20 2024-03-29T11:12:34.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.491+0000 7f12cb59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:34.471 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.491+0000 7f12cb59e640 10 monclient(hunting): authenticate will time out at 935.802980s 2024-03-29T11:12:34.472 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.495+0000 7f12cad9d640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f12cc0aad20 0x7f12cc151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:57167/0 (socket says 172.21.15.170:57167) 2024-03-29T11:12:34.472 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.495+0000 7f12cad9d640 1 -- 172.21.15.170:0/2900255673 learned_addr learned my addr 172.21.15.170:0/2900255673 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:12:34.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 1 -- 172.21.15.170:0/2900255673 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858694718 0 0) 0x7f12cc149ac0 con 0x7f12cc0aad20 2024-03-29T11:12:34.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:12:34.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:34.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:34.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:12:34.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 1 -- 172.21.15.170:0/2900255673 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b8003520 con 0x7f12cc0aad20 2024-03-29T11:12:34.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 1 -- 172.21.15.170:0/2900255673 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 141036868 0 0) 0x7f12b8003520 con 0x7f12cc0aad20 2024-03-29T11:12:34.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 1 -- 172.21.15.170:0/2900255673 >> v1:172.21.15.184:6789/0 conn(0x7f12cc0a9750 legacy=0x7f12cc149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:34.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 1 -- 172.21.15.170:0/2900255673 >> v1:172.21.15.170:6789/0 conn(0x7f12cc0add40 legacy=0x7f12cc143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:34.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:12:34.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 1 monclient: found mon.noname-b 2024-03-29T11:12:34.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:34.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:12:34.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 1 -- 172.21.15.170:0/2900255673 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12cc14a6f0 con 0x7f12cc0aad20 2024-03-29T11:12:34.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 10 monclient: _finish_auth 0 2024-03-29T11:12:34.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:12:34.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:34.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 1 -- 172.21.15.170:0/2900255673 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f12c4001a10 con 0x7f12cc0aad20 2024-03-29T11:12:34.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:34.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:34.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.497+0000 7f12c9d9b640 10 monclient: dump: 2024-03-29T11:12:34.476 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:34.476 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:34.476 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:34.476 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:34.476 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:34.476 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:34.476 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:34.476 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:34.476 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:34.476 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:34.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.498+0000 7f12c9d9b640 1 -- 172.21.15.170:0/2900255673 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12c4001be0 con 0x7f12cc0aad20 2024-03-29T11:12:34.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.498+0000 7f12c9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:34.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.498+0000 7f12c9d9b640 1 -- 172.21.15.170:0/2900255673 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f12c4001ff0 con 0x7f12cc0aad20 2024-03-29T11:12:34.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.498+0000 7f12cb59e640 5 monclient: authenticate success, global_id 4256 2024-03-29T11:12:34.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.498+0000 7f12cb59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:34.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.498+0000 7f12cb59e640 4 set_mon_vals no callback set 2024-03-29T11:12:34.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.498+0000 7f12c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:34.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.498+0000 7f12c9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:34.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.498+0000 7f12c9d9b640 10 monclient: dump: 2024-03-29T11:12:34.478 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:34.478 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:34.478 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:34.478 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:34.479 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:34.479 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:34.479 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:34.479 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:34.479 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:34.479 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:34.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.499+0000 7f12cb59e640 10 monclient: shutdown 2024-03-29T11:12:34.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.499+0000 7f12cb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:34.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.499+0000 7f12cb59e640 1 -- 172.21.15.170:0/2900255673 >> v1:172.21.15.170:6790/0 conn(0x7f12cc0aad20 legacy=0x7f12cc151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:34.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.499+0000 7f12cb59e640 1 -- 172.21.15.170:0/2900255673 shutdown_connections 2024-03-29T11:12:34.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.499+0000 7f12cb59e640 1 -- 172.21.15.170:0/2900255673 >> 172.21.15.170:0/2900255673 conn(0x7f12cc0673e0 msgr2=0x7f12cc0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:34.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.500+0000 7f12cb59e640 1 -- 172.21.15.170:0/2900255673 shutdown_connections 2024-03-29T11:12:34.485 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.504+0000 7f12cb59e640 1 -- 172.21.15.170:0/2900255673 wait complete. 2024-03-29T11:12:34.485 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.505+0000 7f12cb59e640 10 monclient: build_initial_monmap 2024-03-29T11:12:34.485 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.505+0000 7f12cb59e640 10 monclient: monmap: 2024-03-29T11:12:34.485 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:12:34.485 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:34.485 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:12:34.485 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:12:34.486 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:34.486 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:34.486 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:34.486 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:12:34.486 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:12:34.486 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:34.486 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.505+0000 7f12cb59e640 1 librados: starting msgr at 2024-03-29T11:12:34.486 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.505+0000 7f12cb59e640 1 librados: starting objecter 2024-03-29T11:12:34.486 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.506+0000 7f12cb59e640 1 Processor -- start 2024-03-29T11:12:34.486 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.506+0000 7f12cb59e640 1 -- start start 2024-03-29T11:12:34.486 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.506+0000 7f12cb59e640 1 librados: setting wanted keys 2024-03-29T11:12:34.486 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.506+0000 7f12cb59e640 1 librados: calling monclient init 2024-03-29T11:12:34.486 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.506+0000 7f12cb59e640 10 monclient: init 2024-03-29T11:12:34.486 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.506+0000 7f12cb59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:34.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.506+0000 7f12cb59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:12:34.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.506+0000 7f12cb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f12cc0aad20 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:34.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.507+0000 7f12cb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f12cc0add40 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:34.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.507+0000 7f12cb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f12cc06d8c0 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:34.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.507+0000 7f12cb59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12cc1e3ca0 con 0x7f12cc06d8c0 2024-03-29T11:12:34.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.507+0000 7f12cb59e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12cc1e4b20 con 0x7f12cc0add40 2024-03-29T11:12:34.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.507+0000 7f12cb59e640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12cc1e3ef0 con 0x7f12cc0aad20 2024-03-29T11:12:34.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.507+0000 7f12cb59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:34.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.507+0000 7f12cb59e640 10 monclient(hunting): authenticate will time out at 935.818069s 2024-03-29T11:12:34.491 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.513+0000 7f12cad9d640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f12cc06d8c0 0x7f12cc1d9b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:57539/0 (socket says 172.21.15.170:57539) 2024-03-29T11:12:34.491 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.513+0000 7f12cad9d640 1 -- 172.21.15.170:0/2807743204 learned_addr learned my addr 172.21.15.170:0/2807743204 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:12:34.494 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.513+0000 7f12c37fe640 10 client.?.objecter ms_handle_connect 0x7f12cc06d8c0 2024-03-29T11:12:34.494 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.513+0000 7f12c37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:34.494 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.513+0000 7f12c37fe640 10 client.?.objecter ms_handle_connect 0x7f12cc0aad20 2024-03-29T11:12:34.494 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.513+0000 7f12c37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:34.494 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.513+0000 7f12c37fe640 1 -- 172.21.15.170:0/2807743204 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343116877 0 0) 0x7f12cc1e3ca0 con 0x7f12cc06d8c0 2024-03-29T11:12:34.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.513+0000 7f12c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:34.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.513+0000 7f12c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:34.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.513+0000 7f12c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:34.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.513+0000 7f12c37fe640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:12:34.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.513+0000 7f12c37fe640 1 -- 172.21.15.170:0/2807743204 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12ac004eb0 con 0x7f12cc06d8c0 2024-03-29T11:12:34.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.514+0000 7f12c37fe640 1 -- 172.21.15.170:0/2807743204 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3969447177 0 0) 0x7f12ac004eb0 con 0x7f12cc06d8c0 2024-03-29T11:12:34.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.514+0000 7f12c37fe640 1 -- 172.21.15.170:0/2807743204 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12cc1e3ca0 con 0x7f12cc06d8c0 2024-03-29T11:12:34.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.514+0000 7f12c37fe640 1 -- 172.21.15.170:0/2807743204 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f12c4001ef0 con 0x7f12cc06d8c0 2024-03-29T11:12:34.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.514+0000 7f12c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:34.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.514+0000 7f12c37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:34.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.514+0000 7f12c37fe640 10 monclient(hunting): dump: 2024-03-29T11:12:34.495 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:34.495 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:34.495 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:34.495 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:34.496 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:34.496 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:34.496 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:34.496 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:34.496 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:34.496 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:34.496 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.514+0000 7f12c37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:34.496 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.514+0000 7f12c37fe640 1 -- 172.21.15.170:0/2807743204 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465273794 0 0) 0x7f12cc1e3ef0 con 0x7f12cc0aad20 2024-03-29T11:12:34.496 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.514+0000 7f12c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:34.496 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.514+0000 7f12c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:34.496 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.514+0000 7f12c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:34.496 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.514+0000 7f12c37fe640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:12:34.496 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.514+0000 7f12c37fe640 1 -- 172.21.15.170:0/2807743204 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12ac004eb0 con 0x7f12cc0aad20 2024-03-29T11:12:34.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.515+0000 7f12c37fe640 1 -- 172.21.15.170:0/2807743204 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 331068131 0 0) 0x7f12cc1e3ca0 con 0x7f12cc06d8c0 2024-03-29T11:12:34.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.515+0000 7f12c37fe640 1 -- 172.21.15.170:0/2807743204 >> v1:172.21.15.170:6790/0 conn(0x7f12cc0aad20 legacy=0x7f12cc1e4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:34.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.515+0000 7f12c37fe640 1 -- 172.21.15.170:0/2807743204 >> v1:172.21.15.184:6789/0 conn(0x7f12cc0add40 legacy=0x7f12cc070680 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:12:34.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.515+0000 7f12c37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:12:34.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.515+0000 7f12c37fe640 1 monclient: found mon.a 2024-03-29T11:12:34.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.515+0000 7f12c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:34.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.515+0000 7f12c37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:34.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.515+0000 7f12c37fe640 1 -- 172.21.15.170:0/2807743204 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12cc070d90 con 0x7f12cc06d8c0 2024-03-29T11:12:34.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.515+0000 7f12c37fe640 10 monclient: _finish_auth 0 2024-03-29T11:12:34.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.515+0000 7f12c37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:12:34.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.515+0000 7f12c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:34.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.515+0000 7f12c37fe640 1 -- 172.21.15.170:0/2807743204 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12c4004f70 con 0x7f12cc06d8c0 2024-03-29T11:12:34.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.515+0000 7f12c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:34.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.515+0000 7f12c37fe640 1 -- 172.21.15.170:0/2807743204 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f12c4005280 con 0x7f12cc06d8c0 2024-03-29T11:12:34.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.515+0000 7f12c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:34.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.515+0000 7f12c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:34.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.515+0000 7f12c37fe640 10 monclient: dump: 2024-03-29T11:12:34.501 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:34.501 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:34.501 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:34.501 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:34.501 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:34.501 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:34.502 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:34.502 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:34.502 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:34.502 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:34.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.516+0000 7f12c959a640 4 set_mon_vals no callback set 2024-03-29T11:12:34.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.518+0000 7f12cb59e640 5 monclient: authenticate success, global_id 4251 2024-03-29T11:12:34.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.518+0000 7f12cb59e640 10 monclient: _renew_subs 2024-03-29T11:12:34.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.518+0000 7f12cb59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:34.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.518+0000 7f12cb59e640 1 -- 172.21.15.170:0/2807743204 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12cc06fbc0 con 0x7f12cc06d8c0 2024-03-29T11:12:34.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.518+0000 7f12cb59e640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:34.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.518+0000 7f12cb59e640 10 monclient: _renew_subs 2024-03-29T11:12:34.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.518+0000 7f12cb59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:34.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.518+0000 7f12cb59e640 1 -- 172.21.15.170:0/2807743204 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12cc1da290 con 0x7f12cc06d8c0 2024-03-29T11:12:34.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.518+0000 7f12cb59e640 1 librados: init done 2024-03-29T11:12:34.503 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.520+0000 7f12cb59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:34.503 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.520+0000 7f12cb59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:34.503 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.520+0000 7f12cb59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:34.503 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.520+0000 7f12cb59e640 1 -- 172.21.15.170:0/2807743204 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12cc1e4080 con 0x7f12cc06d8c0 2024-03-29T11:12:34.503 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.521+0000 7f12c37fe640 1 -- 172.21.15.170:0/2807743204 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f12c401b250 con 0x7f12cc06d8c0 2024-03-29T11:12:34.503 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.521+0000 7f12c37fe640 1 -- 172.21.15.170:0/2807743204 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f12c4084ef0 con 0x7f12cc06d8c0 2024-03-29T11:12:34.503 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.522+0000 7f12c37fe640 10 client.4251.objecter ms_dispatch 0x7f12cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:34.503 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.522+0000 7f12c37fe640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:34.503 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.522+0000 7f12c37fe640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:34.503 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.522+0000 7f12c37fe640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T11:12:34.504 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.523+0000 7f12ca59c640 10 monclient: get_auth_request con 0x7f12ac06ddf0 auth_method 0 2024-03-29T11:12:34.504 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.524+0000 7f12c37fe640 10 client.4251.objecter ms_handle_connect 0x7f12ac06ddf0 2024-03-29T11:12:34.510 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.532+0000 7f12c37fe640 1 -- 172.21.15.170:0/2807743204 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f12c40507d0 con 0x7f12cc06d8c0 2024-03-29T11:12:34.510 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.532+0000 7f12c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:34.510 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.532+0000 7f12c37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:34.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.854+0000 7f12cb59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:34.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.854+0000 7f12cb59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:34.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.854+0000 7f12cb59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:34.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.854+0000 7f12cb59e640 1 -- 172.21.15.170:0/2807743204 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12cc04f730 con 0x7f12cc06d8c0 2024-03-29T11:12:34.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.856+0000 7f12c37fe640 1 -- 172.21.15.170:0/2807743204 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f12c4052700 con 0x7f12cc06d8c0 2024-03-29T11:12:34.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.856+0000 7f12c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:34.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.856+0000 7f12c37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:34.834 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:12:34.834 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:34.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.860+0000 7f12cb59e640 10 librados: watch_flush enter 2024-03-29T11:12:34.840 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.862+0000 7f12cb59e640 10 librados: watch_flush exit 2024-03-29T11:12:34.840 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.863+0000 7f12cb59e640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T11:12:34.840 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.863+0000 7f12cb59e640 10 client.4251.objecter successfully canceled tick 2024-03-29T11:12:34.841 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.864+0000 7f12cb59e640 1 -- 172.21.15.170:0/2807743204 >> v1:172.21.15.170:6816/3298994444 conn(0x7f12ac06ddf0 legacy=0x7f12ac0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:34.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.864+0000 7f12cb59e640 10 monclient: shutdown 2024-03-29T11:12:34.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.864+0000 7f12cb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:34.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.864+0000 7f12cb59e640 1 -- 172.21.15.170:0/2807743204 >> v1:172.21.15.170:6789/0 conn(0x7f12cc06d8c0 legacy=0x7f12cc1d9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:34.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.864+0000 7f12cb59e640 1 -- 172.21.15.170:0/2807743204 shutdown_connections 2024-03-29T11:12:34.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.864+0000 7f12cb59e640 1 -- 172.21.15.170:0/2807743204 >> 172.21.15.170:0/2807743204 conn(0x7f12cc0673e0 msgr2=0x7f12cc1411a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:34.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.864+0000 7f12cb59e640 1 -- 172.21.15.170:0/2807743204 shutdown_connections 2024-03-29T11:12:34.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.864+0000 7f12cb59e640 1 -- 172.21.15.170:0/2807743204 wait complete. 2024-03-29T11:12:34.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:34.865+0000 7f12cb59e640 1 librados: shutdown 2024-03-29T11:12:34.864 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:41.672 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:41.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.853+0000 7fe7d0a31640 10 monclient: get_monmap_and_config 2024-03-29T11:12:41.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.853+0000 7fe7d0a31640 10 monclient: build_initial_monmap 2024-03-29T11:12:41.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.853+0000 7fe7d0a31640 10 monclient: monmap: 2024-03-29T11:12:41.833 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:12:41.833 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:41.833 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:12:41.854268+0000 2024-03-29T11:12:41.833 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:12:41.854268+0000 2024-03-29T11:12:41.833 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:41.833 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:41.833 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:41.833 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:12:41.833 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:12:41.833 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:41.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.859+0000 7fe7d0a31640 1 Processor -- start 2024-03-29T11:12:41.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.859+0000 7fe7d0a31640 1 -- start start 2024-03-29T11:12:41.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.859+0000 7fe7d0a31640 10 monclient: init 2024-03-29T11:12:41.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.860+0000 7fe7d0a31640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:41.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.860+0000 7fe7d0a31640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:41.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.860+0000 7fe7d0a31640 10 monclient(hunting): picked mon.noname-a con 0x7fe7c4034850 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:41.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.860+0000 7fe7d0a31640 10 monclient(hunting): picked mon.noname-b con 0x7fe7c402a640 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:41.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.860+0000 7fe7d0a31640 10 monclient(hunting): picked mon.noname-c con 0x7fe7c40327d0 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:41.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.860+0000 7fe7d0a31640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c402c820 con 0x7fe7c4034850 2024-03-29T11:12:41.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.860+0000 7fe7d0a31640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c402a120 con 0x7fe7c40327d0 2024-03-29T11:12:41.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.860+0000 7fe7d0a31640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c40c75b0 con 0x7fe7c402a640 2024-03-29T11:12:41.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.860+0000 7fe7d0a31640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:41.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.860+0000 7fe7d0a31640 10 monclient(hunting): authenticate will time out at 943.171909s 2024-03-29T11:12:41.841 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.863+0000 7fe7cb7fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fe7c4034850 0x7fe7c40c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:50507/0 (socket says 172.21.15.170:50507) 2024-03-29T11:12:41.841 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.864+0000 7fe7cb7fe640 1 -- 172.21.15.170:0/1365081837 learned_addr learned my addr 172.21.15.170:0/1365081837 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:12:41.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.864+0000 7fe7ca7fc640 1 -- 172.21.15.170:0/1365081837 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346006911 0 0) 0x7fe7c40c75b0 con 0x7fe7c402a640 2024-03-29T11:12:41.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.864+0000 7fe7ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:41.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.864+0000 7fe7ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:41.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.864+0000 7fe7ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:41.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.864+0000 7fe7ca7fc640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:12:41.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.864+0000 7fe7ca7fc640 1 -- 172.21.15.170:0/1365081837 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7bc003520 con 0x7fe7c402a640 2024-03-29T11:12:41.847 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.866+0000 7fe7ca7fc640 1 -- 172.21.15.170:0/1365081837 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390027140 0 0) 0x7fe7c402c820 con 0x7fe7c4034850 2024-03-29T11:12:41.847 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.866+0000 7fe7ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:41.847 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.866+0000 7fe7ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:41.847 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.866+0000 7fe7ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:41.847 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.866+0000 7fe7ca7fc640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:12:41.847 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.866+0000 7fe7ca7fc640 1 -- 172.21.15.170:0/1365081837 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c40c75b0 con 0x7fe7c4034850 2024-03-29T11:12:41.847 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.866+0000 7fe7ca7fc640 1 -- 172.21.15.170:0/1365081837 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 18642403 0 0) 0x7fe7bc003520 con 0x7fe7c402a640 2024-03-29T11:12:41.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.866+0000 7fe7ca7fc640 1 -- 172.21.15.170:0/1365081837 >> v1:172.21.15.184:6789/0 conn(0x7fe7c40327d0 legacy=0x7fe7c40cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:41.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.866+0000 7fe7ca7fc640 1 -- 172.21.15.170:0/1365081837 >> v1:172.21.15.170:6789/0 conn(0x7fe7c4034850 legacy=0x7fe7c40c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:41.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.867+0000 7fe7ca7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:12:41.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.867+0000 7fe7ca7fc640 1 monclient: found mon.noname-b 2024-03-29T11:12:41.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.867+0000 7fe7ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:41.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.867+0000 7fe7ca7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:12:41.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.867+0000 7fe7ca7fc640 1 -- 172.21.15.170:0/1365081837 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c4033110 con 0x7fe7c402a640 2024-03-29T11:12:41.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.867+0000 7fe7ca7fc640 10 monclient: _finish_auth 0 2024-03-29T11:12:41.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.867+0000 7fe7ca7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:12:41.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.867+0000 7fe7ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:41.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.867+0000 7fe7ca7fc640 1 -- 172.21.15.170:0/1365081837 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe7b8001690 con 0x7fe7c402a640 2024-03-29T11:12:41.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.867+0000 7fe7ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:41.849 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.867+0000 7fe7ca7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:41.849 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.867+0000 7fe7ca7fc640 10 monclient: dump: 2024-03-29T11:12:41.849 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:41.849 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:41.849 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:41.849 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:41.849 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:41.849 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:41.849 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:41.849 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:41.849 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:41.849 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:41.850 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.867+0000 7fe7d0a31640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:12:41.850 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.867+0000 7fe7d0a31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:41.854 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.871+0000 7fe7ca7fc640 1 -- 172.21.15.170:0/1365081837 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7b80018b0 con 0x7fe7c402a640 2024-03-29T11:12:41.854 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.871+0000 7fe7ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:41.854 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.871+0000 7fe7ca7fc640 1 -- 172.21.15.170:0/1365081837 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe7b8001c40 con 0x7fe7c402a640 2024-03-29T11:12:41.854 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.871+0000 7fe7d0a31640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:41.854 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.871+0000 7fe7d0a31640 4 set_mon_vals no callback set 2024-03-29T11:12:41.854 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.871+0000 7fe7ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:41.854 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.871+0000 7fe7ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:41.855 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.871+0000 7fe7ca7fc640 10 monclient: dump: 2024-03-29T11:12:41.855 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:41.855 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:41.855 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:41.855 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:41.855 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:41.855 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:41.855 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:41.855 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:41.855 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:41.855 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:41.855 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.872+0000 7fe7d0a31640 10 monclient: shutdown 2024-03-29T11:12:41.856 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.872+0000 7fe7d0a31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:41.856 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.872+0000 7fe7d0a31640 1 -- 172.21.15.170:0/1365081837 >> v1:172.21.15.170:6790/0 conn(0x7fe7c402a640 legacy=0x7fe7c40320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:41.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.872+0000 7fe7d0a31640 1 -- 172.21.15.170:0/1365081837 shutdown_connections 2024-03-29T11:12:41.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.872+0000 7fe7d0a31640 1 -- 172.21.15.170:0/1365081837 >> 172.21.15.170:0/1365081837 conn(0x7fe7c40058f0 msgr2=0x7fe7c4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:41.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.872+0000 7fe7d0a31640 1 -- 172.21.15.170:0/1365081837 shutdown_connections 2024-03-29T11:12:41.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.872+0000 7fe7d0a31640 1 -- 172.21.15.170:0/1365081837 wait complete. 2024-03-29T11:12:41.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.873+0000 7fe7d0a31640 10 monclient: build_initial_monmap 2024-03-29T11:12:41.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.873+0000 7fe7d0a31640 10 monclient: monmap: 2024-03-29T11:12:41.858 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:12:41.858 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:41.858 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:12:41.859 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:12:41.859 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:41.859 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:41.859 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:41.859 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:12:41.859 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:12:41.859 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:41.859 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.873+0000 7fe7d0a31640 1 librados: starting msgr at 2024-03-29T11:12:41.859 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.873+0000 7fe7d0a31640 1 librados: starting objecter 2024-03-29T11:12:41.859 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.874+0000 7fe7d0a31640 1 Processor -- start 2024-03-29T11:12:41.860 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.874+0000 7fe7d0a31640 1 -- start start 2024-03-29T11:12:41.860 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.874+0000 7fe7d0a31640 1 librados: setting wanted keys 2024-03-29T11:12:41.860 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.874+0000 7fe7d0a31640 1 librados: calling monclient init 2024-03-29T11:12:41.860 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.874+0000 7fe7d0a31640 10 monclient: init 2024-03-29T11:12:41.860 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.875+0000 7fe7d0a31640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:41.860 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.875+0000 7fe7d0a31640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:41.860 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.875+0000 7fe7d0a31640 10 monclient(hunting): picked mon.noname-b con 0x7fe7c40327d0 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:41.860 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.875+0000 7fe7d0a31640 10 monclient(hunting): picked mon.noname-a con 0x7fe7c4034850 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:41.860 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.875+0000 7fe7d0a31640 10 monclient(hunting): picked mon.noname-c con 0x7fe7c40c4050 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:41.861 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.875+0000 7fe7d0a31640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c402f8c0 con 0x7fe7c4034850 2024-03-29T11:12:41.861 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.875+0000 7fe7d0a31640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c4164820 con 0x7fe7c40327d0 2024-03-29T11:12:41.861 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.875+0000 7fe7d0a31640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c402dfd0 con 0x7fe7c40c4050 2024-03-29T11:12:41.861 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.875+0000 7fe7d0a31640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:41.861 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.875+0000 7fe7d0a31640 10 monclient(hunting): authenticate will time out at 943.186571s 2024-03-29T11:12:41.872 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.883+0000 7fe7cb7fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fe7c40327d0 0x7fe7c4164110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:57405/0 (socket says 172.21.15.170:57405) 2024-03-29T11:12:41.872 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.883+0000 7fe7cb7fe640 1 -- 172.21.15.170:0/2055816776 learned_addr learned my addr 172.21.15.170:0/2055816776 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:12:41.872 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.883+0000 7fe7b7fff640 10 client.?.objecter ms_handle_connect 0x7fe7c4034850 2024-03-29T11:12:41.873 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.883+0000 7fe7b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:41.873 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.883+0000 7fe7b7fff640 1 -- 172.21.15.170:0/2055816776 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193007630 0 0) 0x7fe7c402f8c0 con 0x7fe7c4034850 2024-03-29T11:12:41.873 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.883+0000 7fe7b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:41.873 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.883+0000 7fe7b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:41.873 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.883+0000 7fe7b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:41.873 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.883+0000 7fe7b7fff640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:12:41.873 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.883+0000 7fe7b7fff640 1 -- 172.21.15.170:0/2055816776 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7b0004eb0 con 0x7fe7c4034850 2024-03-29T11:12:41.873 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.884+0000 7fe7b7fff640 10 client.?.objecter ms_handle_connect 0x7fe7c40327d0 2024-03-29T11:12:41.873 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.884+0000 7fe7b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:41.874 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.884+0000 7fe7b7fff640 1 -- 172.21.15.170:0/2055816776 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3164927679 0 0) 0x7fe7b0004eb0 con 0x7fe7c4034850 2024-03-29T11:12:41.874 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.884+0000 7fe7b7fff640 1 -- 172.21.15.170:0/2055816776 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7c402f8c0 con 0x7fe7c4034850 2024-03-29T11:12:41.874 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.884+0000 7fe7b7fff640 1 -- 172.21.15.170:0/2055816776 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe7b80015d0 con 0x7fe7c4034850 2024-03-29T11:12:41.874 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.884+0000 7fe7b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:41.874 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.884+0000 7fe7b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:41.874 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.884+0000 7fe7b7fff640 10 monclient(hunting): dump: 2024-03-29T11:12:41.874 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:41.875 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:41.875 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:41.875 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:41.875 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:41.875 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:41.875 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:41.875 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:41.875 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:41.876 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:41.876 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.884+0000 7fe7b7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:41.876 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.885+0000 7fe7b7fff640 10 client.?.objecter ms_handle_connect 0x7fe7c40c4050 2024-03-29T11:12:41.876 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.885+0000 7fe7b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:41.876 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.885+0000 7fe7b7fff640 1 -- 172.21.15.170:0/2055816776 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835757935 0 0) 0x7fe7c4164820 con 0x7fe7c40327d0 2024-03-29T11:12:41.876 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.885+0000 7fe7b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:41.876 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.886+0000 7fe7b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:41.876 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.886+0000 7fe7b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:41.876 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.886+0000 7fe7b7fff640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:12:41.876 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.886+0000 7fe7b7fff640 1 -- 172.21.15.170:0/2055816776 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7b0004eb0 con 0x7fe7c40327d0 2024-03-29T11:12:41.876 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.886+0000 7fe7b7fff640 1 -- 172.21.15.170:0/2055816776 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318511017 0 0) 0x7fe7c402dfd0 con 0x7fe7c40c4050 2024-03-29T11:12:41.876 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.886+0000 7fe7b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:41.877 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.886+0000 7fe7b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:41.877 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.886+0000 7fe7b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:41.877 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.886+0000 7fe7b7fff640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:12:41.877 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.886+0000 7fe7b7fff640 1 -- 172.21.15.170:0/2055816776 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c4164820 con 0x7fe7c40c4050 2024-03-29T11:12:41.877 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.886+0000 7fe7b7fff640 1 -- 172.21.15.170:0/2055816776 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3158986426 0 0) 0x7fe7c402f8c0 con 0x7fe7c4034850 2024-03-29T11:12:41.877 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.886+0000 7fe7b7fff640 1 -- 172.21.15.170:0/2055816776 >> v1:172.21.15.170:6790/0 conn(0x7fe7c40c4050 legacy=0x7fe7c4031020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:41.877 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.886+0000 7fe7b7fff640 1 -- 172.21.15.170:0/2055816776 >> v1:172.21.15.184:6789/0 conn(0x7fe7c40327d0 legacy=0x7fe7c4164110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:41.877 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.886+0000 7fe7b7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:12:41.877 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.887+0000 7fe7b7fff640 1 monclient: found mon.a 2024-03-29T11:12:41.877 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.887+0000 7fe7b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:41.877 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.887+0000 7fe7b7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:41.877 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.887+0000 7fe7b7fff640 1 -- 172.21.15.170:0/2055816776 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c402dba0 con 0x7fe7c4034850 2024-03-29T11:12:41.878 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.887+0000 7fe7b7fff640 10 monclient: _finish_auth 0 2024-03-29T11:12:41.878 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.887+0000 7fe7b7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:12:41.878 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.887+0000 7fe7b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:41.878 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.887+0000 7fe7d0a31640 5 monclient: authenticate success, global_id 4257 2024-03-29T11:12:41.879 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.887+0000 7fe7d0a31640 10 monclient: _renew_subs 2024-03-29T11:12:41.879 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.887+0000 7fe7d0a31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:41.879 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.887+0000 7fe7d0a31640 1 -- 172.21.15.170:0/2055816776 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c402e4f0 con 0x7fe7c4034850 2024-03-29T11:12:41.879 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.887+0000 7fe7d0a31640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:41.879 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.887+0000 7fe7d0a31640 10 monclient: _renew_subs 2024-03-29T11:12:41.879 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.887+0000 7fe7d0a31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:41.879 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.887+0000 7fe7d0a31640 1 -- 172.21.15.170:0/2055816776 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c4031730 con 0x7fe7c4034850 2024-03-29T11:12:41.879 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.887+0000 7fe7d0a31640 1 librados: init done 2024-03-29T11:12:41.879 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.888+0000 7fe7b7fff640 1 -- 172.21.15.170:0/2055816776 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7b80038a0 con 0x7fe7c4034850 2024-03-29T11:12:41.880 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.888+0000 7fe7b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:41.880 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.888+0000 7fe7b7fff640 1 -- 172.21.15.170:0/2055816776 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe7b8003bb0 con 0x7fe7c4034850 2024-03-29T11:12:41.880 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.888+0000 7fe7b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:41.880 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.888+0000 7fe7b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:41.880 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.888+0000 7fe7b7fff640 10 monclient: dump: 2024-03-29T11:12:41.880 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:41.880 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:41.880 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:41.880 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:41.880 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:41.880 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:41.880 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:41.880 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:41.881 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:41.881 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:41.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.888+0000 7fe7c9ffb640 4 set_mon_vals no callback set 2024-03-29T11:12:41.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.889+0000 7fe7d0a31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:41.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.889+0000 7fe7d0a31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:41.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.889+0000 7fe7d0a31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:41.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.889+0000 7fe7d0a31640 1 -- 172.21.15.170:0/2055816776 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7ac005080 con 0x7fe7c4034850 2024-03-29T11:12:41.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.892+0000 7fe7b7fff640 1 -- 172.21.15.170:0/2055816776 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fe7b8019b80 con 0x7fe7c4034850 2024-03-29T11:12:41.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.893+0000 7fe7b7fff640 1 -- 172.21.15.170:0/2055816776 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fe7b80837d0 con 0x7fe7c4034850 2024-03-29T11:12:41.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.894+0000 7fe7b7fff640 10 client.4257.objecter ms_dispatch 0x7fe7cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:41.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.894+0000 7fe7b7fff640 3 client.4257.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:41.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.894+0000 7fe7b7fff640 3 client.4257.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:41.882 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.894+0000 7fe7b7fff640 20 client.4257.objecter dump_active .. 0 homeless 2024-03-29T11:12:41.882 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.894+0000 7fe7cb7fe640 10 monclient: get_auth_request con 0x7fe7b006db40 auth_method 0 2024-03-29T11:12:41.882 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.894+0000 7fe7b7fff640 10 client.4257.objecter ms_handle_connect 0x7fe7b006db40 2024-03-29T11:12:41.883 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.902+0000 7fe7b7fff640 1 -- 172.21.15.170:0/2055816776 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe7b804f110 con 0x7fe7c4034850 2024-03-29T11:12:41.883 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.902+0000 7fe7b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:41.883 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:41.902+0000 7fe7b7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:42.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.230+0000 7fe7d0a31640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:42.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.230+0000 7fe7d0a31640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:42.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.230+0000 7fe7d0a31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:42.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.230+0000 7fe7d0a31640 1 -- 172.21.15.170:0/2055816776 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7ac005330 con 0x7fe7c4034850 2024-03-29T11:12:42.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.231+0000 7fe7b7fff640 1 -- 172.21.15.170:0/2055816776 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fe7b8051040 con 0x7fe7c4034850 2024-03-29T11:12:42.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.231+0000 7fe7b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:42.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.231+0000 7fe7b7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:42.209 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:12:42.209 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:42.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.235+0000 7fe7d0a31640 10 librados: watch_flush enter 2024-03-29T11:12:42.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.235+0000 7fe7d0a31640 10 librados: watch_flush exit 2024-03-29T11:12:42.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.235+0000 7fe7d0a31640 20 client.4257.objecter shutdown clearing up homeless session... 2024-03-29T11:12:42.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.235+0000 7fe7d0a31640 10 client.4257.objecter successfully canceled tick 2024-03-29T11:12:42.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.235+0000 7fe7d0a31640 1 -- 172.21.15.170:0/2055816776 >> v1:172.21.15.170:6816/3298994444 conn(0x7fe7b006db40 legacy=0x7fe7b0070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.235+0000 7fe7d0a31640 10 monclient: shutdown 2024-03-29T11:12:42.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.235+0000 7fe7d0a31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:42.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.236+0000 7fe7d0a31640 1 -- 172.21.15.170:0/2055816776 >> v1:172.21.15.170:6789/0 conn(0x7fe7c4034850 legacy=0x7fe7c402d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.236+0000 7fe7d0a31640 1 -- 172.21.15.170:0/2055816776 shutdown_connections 2024-03-29T11:12:42.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.236+0000 7fe7d0a31640 1 -- 172.21.15.170:0/2055816776 >> 172.21.15.170:0/2055816776 conn(0x7fe7c40058f0 msgr2=0x7fe7c4038110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:42.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.236+0000 7fe7d0a31640 1 -- 172.21.15.170:0/2055816776 shutdown_connections 2024-03-29T11:12:42.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.236+0000 7fe7d0a31640 1 -- 172.21.15.170:0/2055816776 wait complete. 2024-03-29T11:12:42.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:42.236+0000 7fe7d0a31640 1 librados: shutdown 2024-03-29T11:12:42.233 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:49.040 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:49.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.215+0000 7f6244bd0640 10 monclient: get_monmap_and_config 2024-03-29T11:12:49.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.215+0000 7f6244bd0640 10 monclient: build_initial_monmap 2024-03-29T11:12:49.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.215+0000 7f6244bd0640 10 monclient: monmap: 2024-03-29T11:12:49.194 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:12:49.194 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:49.194 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:12:49.216112+0000 2024-03-29T11:12:49.194 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:12:49.216112+0000 2024-03-29T11:12:49.194 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:49.195 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:49.195 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:49.195 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:12:49.195 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:12:49.195 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:49.196 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.217+0000 7f6244bd0640 1 Processor -- start 2024-03-29T11:12:49.196 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.217+0000 7f6244bd0640 1 -- start start 2024-03-29T11:12:49.197 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.217+0000 7f6244bd0640 10 monclient: init 2024-03-29T11:12:49.197 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.218+0000 7f6244bd0640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:49.197 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.218+0000 7f6244bd0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:12:49.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.218+0000 7f6244bd0640 10 monclient(hunting): picked mon.noname-b con 0x7f62400b4100 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:49.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.218+0000 7f6244bd0640 10 monclient(hunting): picked mon.noname-c con 0x7f624013ee90 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:49.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.219+0000 7f6244bd0640 10 monclient(hunting): picked mon.noname-a con 0x7f62400af900 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:49.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.219+0000 7f6244bd0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62400afd10 con 0x7f62400af900 2024-03-29T11:12:49.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.219+0000 7f6244bd0640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62400af6f0 con 0x7f624013ee90 2024-03-29T11:12:49.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.219+0000 7f6244bd0640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62400aac80 con 0x7f62400b4100 2024-03-29T11:12:49.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.219+0000 7f6244bd0640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:49.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.219+0000 7f6244bd0640 10 monclient(hunting): authenticate will time out at 950.530070s 2024-03-29T11:12:49.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.222+0000 7f623ed9d640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f62400af900 0x7f624014be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:42199/0 (socket says 172.21.15.170:42199) 2024-03-29T11:12:49.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.222+0000 7f623ed9d640 1 -- 172.21.15.170:0/3809893466 learned_addr learned my addr 172.21.15.170:0/3809893466 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:12:49.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.224+0000 7f623d59a640 1 -- 172.21.15.170:0/3809893466 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927462591 0 0) 0x7f62400afd10 con 0x7f62400af900 2024-03-29T11:12:49.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.224+0000 7f623d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:49.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.224+0000 7f623d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:49.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.224+0000 7f623d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:49.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.224+0000 7f623d59a640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:12:49.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.224+0000 7f623d59a640 1 -- 172.21.15.170:0/3809893466 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6228003520 con 0x7f62400af900 2024-03-29T11:12:49.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 1 -- 172.21.15.170:0/3809893466 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650686496 0 0) 0x7f62400af6f0 con 0x7f624013ee90 2024-03-29T11:12:49.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:49.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:49.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:49.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:12:49.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 1 -- 172.21.15.170:0/3809893466 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62400afd10 con 0x7f624013ee90 2024-03-29T11:12:49.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 1 -- 172.21.15.170:0/3809893466 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 946836420 0 0) 0x7f6228003520 con 0x7f62400af900 2024-03-29T11:12:49.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 1 -- 172.21.15.170:0/3809893466 >> v1:172.21.15.170:6790/0 conn(0x7f62400b4100 legacy=0x7f62400adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:49.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 1 -- 172.21.15.170:0/3809893466 >> v1:172.21.15.184:6789/0 conn(0x7f624013ee90 legacy=0x7f624013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:49.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:12:49.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 1 monclient: found mon.noname-a 2024-03-29T11:12:49.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:49.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:12:49.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 1 -- 172.21.15.170:0/3809893466 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62400aaf60 con 0x7f62400af900 2024-03-29T11:12:49.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 10 monclient: _finish_auth 0 2024-03-29T11:12:49.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:12:49.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:49.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f6244bd0640 5 monclient: authenticate success, global_id 4260 2024-03-29T11:12:49.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f6244bd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:49.206 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 1 -- 172.21.15.170:0/3809893466 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f6234001690 con 0x7f62400af900 2024-03-29T11:12:49.206 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:49.206 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:49.206 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f623d59a640 10 monclient: dump: 2024-03-29T11:12:49.206 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:49.206 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:49.206 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:49.206 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:49.206 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:49.206 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:49.206 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:49.206 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:49.206 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:49.206 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:49.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.227+0000 7f6244bd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:49.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.228+0000 7f623d59a640 1 -- 172.21.15.170:0/3809893466 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62340017f0 con 0x7f62400af900 2024-03-29T11:12:49.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.228+0000 7f623d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:49.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.228+0000 7f623d59a640 1 -- 172.21.15.170:0/3809893466 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f6234001c00 con 0x7f62400af900 2024-03-29T11:12:49.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.228+0000 7f623d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:49.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.228+0000 7f623d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:49.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.228+0000 7f623d59a640 10 monclient: dump: 2024-03-29T11:12:49.207 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:49.208 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:49.208 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:49.208 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:49.208 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:49.208 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:49.208 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:49.208 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:49.208 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:49.208 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:49.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.231+0000 7f6244bd0640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:49.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.231+0000 7f6244bd0640 4 set_mon_vals no callback set 2024-03-29T11:12:49.210 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.232+0000 7f6244bd0640 10 monclient: shutdown 2024-03-29T11:12:49.211 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.232+0000 7f6244bd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:49.211 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.232+0000 7f6244bd0640 1 -- 172.21.15.170:0/3809893466 >> v1:172.21.15.170:6789/0 conn(0x7f62400af900 legacy=0x7f624014be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:49.211 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.233+0000 7f6244bd0640 1 -- 172.21.15.170:0/3809893466 shutdown_connections 2024-03-29T11:12:49.211 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.233+0000 7f6244bd0640 1 -- 172.21.15.170:0/3809893466 >> 172.21.15.170:0/3809893466 conn(0x7f62400b35e0 msgr2=0x7f62400b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:49.211 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.233+0000 7f6244bd0640 1 -- 172.21.15.170:0/3809893466 shutdown_connections 2024-03-29T11:12:49.211 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.233+0000 7f6244bd0640 1 -- 172.21.15.170:0/3809893466 wait complete. 2024-03-29T11:12:49.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.234+0000 7f6244bd0640 10 monclient: build_initial_monmap 2024-03-29T11:12:49.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.234+0000 7f6244bd0640 10 monclient: monmap: 2024-03-29T11:12:49.213 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:12:49.213 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:49.213 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:12:49.213 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:12:49.213 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:49.213 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:49.213 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:49.213 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:12:49.213 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:12:49.213 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:49.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.235+0000 7f6244bd0640 1 librados: starting msgr at 2024-03-29T11:12:49.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.235+0000 7f6244bd0640 1 librados: starting objecter 2024-03-29T11:12:49.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.235+0000 7f6244bd0640 1 Processor -- start 2024-03-29T11:12:49.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.235+0000 7f6244bd0640 1 -- start start 2024-03-29T11:12:49.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.235+0000 7f6244bd0640 1 librados: setting wanted keys 2024-03-29T11:12:49.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.235+0000 7f6244bd0640 1 librados: calling monclient init 2024-03-29T11:12:49.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.235+0000 7f6244bd0640 10 monclient: init 2024-03-29T11:12:49.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.236+0000 7f6244bd0640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:49.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.236+0000 7f6244bd0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:49.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.236+0000 7f6244bd0640 10 monclient(hunting): picked mon.noname-a con 0x7f62400af900 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:49.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.236+0000 7f6244bd0640 10 monclient(hunting): picked mon.noname-b con 0x7f62400b4100 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:49.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.236+0000 7f6244bd0640 10 monclient(hunting): picked mon.noname-c con 0x7f624013ee90 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:49.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.236+0000 7f6244bd0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6240070350 con 0x7f62400af900 2024-03-29T11:12:49.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.236+0000 7f6244bd0640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6240141db0 con 0x7f62400b4100 2024-03-29T11:12:49.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.236+0000 7f6244bd0640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6240142090 con 0x7f624013ee90 2024-03-29T11:12:49.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.236+0000 7f6244bd0640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:49.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.236+0000 7f6244bd0640 10 monclient(hunting): authenticate will time out at 950.547210s 2024-03-29T11:12:49.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.243+0000 7f623dd9b640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f62400b4100 0x7f624006ea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:45069/0 (socket says 172.21.15.170:45069) 2024-03-29T11:12:49.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.243+0000 7f623dd9b640 1 -- 172.21.15.170:0/3374989685 learned_addr learned my addr 172.21.15.170:0/3374989685 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:12:49.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.243+0000 7f622effd640 10 client.?.objecter ms_handle_connect 0x7f62400af900 2024-03-29T11:12:49.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.243+0000 7f622effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:49.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.243+0000 7f622effd640 10 client.?.objecter ms_handle_connect 0x7f624013ee90 2024-03-29T11:12:49.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.243+0000 7f622effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:49.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.243+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742024345 0 0) 0x7f6240070350 con 0x7f62400af900 2024-03-29T11:12:49.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.243+0000 7f622effd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:49.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.243+0000 7f622effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:49.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.243+0000 7f622effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:49.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.243+0000 7f622effd640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:12:49.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.243+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6220004eb0 con 0x7f62400af900 2024-03-29T11:12:49.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 10 client.?.objecter ms_handle_connect 0x7f62400b4100 2024-03-29T11:12:49.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:49.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762270364 0 0) 0x7f6240142090 con 0x7f624013ee90 2024-03-29T11:12:49.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:49.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:49.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:49.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:12:49.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6240070350 con 0x7f624013ee90 2024-03-29T11:12:49.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881345612 0 0) 0x7f6240141db0 con 0x7f62400b4100 2024-03-29T11:12:49.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:49.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:49.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:49.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:12:49.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6240142090 con 0x7f62400b4100 2024-03-29T11:12:49.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1358814304 0 0) 0x7f6220004eb0 con 0x7f62400af900 2024-03-29T11:12:49.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6240141db0 con 0x7f62400af900 2024-03-29T11:12:49.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f6238001c40 con 0x7f62400af900 2024-03-29T11:12:49.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:49.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:49.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 10 monclient(hunting): dump: 2024-03-29T11:12:49.231 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:49.231 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:49.231 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:49.231 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:49.231 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:49.231 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:49.231 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:49.231 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:49.231 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:49.231 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:49.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:49.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2474445182 0 0) 0x7f6240070350 con 0x7f624013ee90 2024-03-29T11:12:49.247 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6220004eb0 con 0x7f624013ee90 2024-03-29T11:12:49.247 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3959245213 0 0) 0x7f6240141db0 con 0x7f62400af900 2024-03-29T11:12:49.247 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 >> v1:172.21.15.170:6790/0 conn(0x7f624013ee90 legacy=0x7f62401e4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:49.248 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 >> v1:172.21.15.184:6789/0 conn(0x7f62400b4100 legacy=0x7f624006ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:49.248 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:49.248 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.244+0000 7f622effd640 1 monclient: found mon.a 2024-03-29T11:12:49.248 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.245+0000 7f622effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:49.248 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.245+0000 7f622effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:49.248 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.245+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6240137c50 con 0x7f62400af900 2024-03-29T11:12:49.248 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.245+0000 7f622effd640 10 monclient: _finish_auth 0 2024-03-29T11:12:49.248 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.245+0000 7f622effd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:49.248 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.245+0000 7f622effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:49.248 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.245+0000 7f6244bd0640 5 monclient: authenticate success, global_id 4263 2024-03-29T11:12:49.248 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.245+0000 7f6244bd0640 10 monclient: _renew_subs 2024-03-29T11:12:49.249 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.245+0000 7f6244bd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:49.249 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.245+0000 7f6244bd0640 1 -- 172.21.15.170:0/3374989685 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6240072600 con 0x7f62400af900 2024-03-29T11:12:49.249 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.245+0000 7f6244bd0640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:49.250 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.245+0000 7f6244bd0640 10 monclient: _renew_subs 2024-03-29T11:12:49.250 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.245+0000 7f6244bd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:49.250 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.245+0000 7f6244bd0640 1 -- 172.21.15.170:0/3374989685 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f624006f300 con 0x7f62400af900 2024-03-29T11:12:49.250 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.245+0000 7f6244bd0640 1 librados: init done 2024-03-29T11:12:49.250 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.246+0000 7f6244bd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:49.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.246+0000 7f6244bd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:49.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.246+0000 7f6244bd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:49.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.246+0000 7f6244bd0640 1 -- 172.21.15.170:0/3374989685 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6240075f10 con 0x7f62400af900 2024-03-29T11:12:49.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.248+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6238003970 con 0x7f62400af900 2024-03-29T11:12:49.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.248+0000 7f622effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:49.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.248+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f6238003c80 con 0x7f62400af900 2024-03-29T11:12:49.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.248+0000 7f622effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:49.252 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.248+0000 7f622effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:49.252 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.248+0000 7f622effd640 10 monclient: dump: 2024-03-29T11:12:49.252 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:49.252 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:49.252 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:49.252 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:49.252 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:49.252 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:49.252 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:49.252 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:49.252 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:49.252 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:49.252 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.248+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f6238019c50 con 0x7f62400af900 2024-03-29T11:12:49.252 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.249+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f6238083bc0 con 0x7f62400af900 2024-03-29T11:12:49.253 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.249+0000 7f622effd640 10 client.4263.objecter ms_dispatch 0x7f6240000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:49.253 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.249+0000 7f622effd640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:49.253 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.249+0000 7f622effd640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:49.253 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.249+0000 7f622effd640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-29T11:12:49.253 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.249+0000 7f623cd99640 4 set_mon_vals no callback set 2024-03-29T11:12:49.253 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.249+0000 7f623dd9b640 10 monclient: get_auth_request con 0x7f622006ddf0 auth_method 0 2024-03-29T11:12:49.253 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.249+0000 7f622effd640 10 client.4263.objecter ms_handle_connect 0x7f622006ddf0 2024-03-29T11:12:49.254 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.258+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f623804f520 con 0x7f62400af900 2024-03-29T11:12:49.254 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.259+0000 7f622effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:49.254 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.259+0000 7f622effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:49.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.584+0000 7f6244bd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:49.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.584+0000 7f6244bd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:49.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.584+0000 7f6244bd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:12:49.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.584+0000 7f6244bd0640 1 -- 172.21.15.170:0/3374989685 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f624006ab70 con 0x7f62400af900 2024-03-29T11:12:49.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.585+0000 7f622effd640 1 -- 172.21.15.170:0/3374989685 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f6238051450 con 0x7f62400af900 2024-03-29T11:12:49.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.585+0000 7f622effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:49.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.585+0000 7f622effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:49.565 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:12:49.565 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:49.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.590+0000 7f6244bd0640 10 librados: watch_flush enter 2024-03-29T11:12:49.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.590+0000 7f6244bd0640 10 librados: watch_flush exit 2024-03-29T11:12:49.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.590+0000 7f6244bd0640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-29T11:12:49.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.590+0000 7f6244bd0640 10 client.4263.objecter successfully canceled tick 2024-03-29T11:12:49.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.590+0000 7f6244bd0640 1 -- 172.21.15.170:0/3374989685 >> v1:172.21.15.170:6816/3298994444 conn(0x7f622006ddf0 legacy=0x7f62200702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:49.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.590+0000 7f6244bd0640 10 monclient: shutdown 2024-03-29T11:12:49.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.590+0000 7f6244bd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:49.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.590+0000 7f6244bd0640 1 -- 172.21.15.170:0/3374989685 >> v1:172.21.15.170:6789/0 conn(0x7f62400af900 legacy=0x7f6240071ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:49.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.590+0000 7f6244bd0640 1 -- 172.21.15.170:0/3374989685 shutdown_connections 2024-03-29T11:12:49.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.590+0000 7f6244bd0640 1 -- 172.21.15.170:0/3374989685 >> 172.21.15.170:0/3374989685 conn(0x7f62400b35e0 msgr2=0x7f62401428b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:49.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.592+0000 7f6244bd0640 1 -- 172.21.15.170:0/3374989685 shutdown_connections 2024-03-29T11:12:49.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.592+0000 7f6244bd0640 1 -- 172.21.15.170:0/3374989685 wait complete. 2024-03-29T11:12:49.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:49.592+0000 7f6244bd0640 1 librados: shutdown 2024-03-29T11:12:49.589 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:56.401 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:56.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.577+0000 7fcb4cbcf640 10 monclient: get_monmap_and_config 2024-03-29T11:12:56.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.577+0000 7fcb4cbcf640 10 monclient: build_initial_monmap 2024-03-29T11:12:56.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.577+0000 7fcb4cbcf640 10 monclient: monmap: 2024-03-29T11:12:56.556 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:12:56.556 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:56.556 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:12:56.578991+0000 2024-03-29T11:12:56.556 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:12:56.578991+0000 2024-03-29T11:12:56.556 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:56.556 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:56.557 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:56.557 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:12:56.557 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:12:56.557 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:56.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.578+0000 7fcb4cbcf640 1 Processor -- start 2024-03-29T11:12:56.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.578+0000 7fcb4cbcf640 1 -- start start 2024-03-29T11:12:56.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.578+0000 7fcb4cbcf640 10 monclient: init 2024-03-29T11:12:56.560 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.579+0000 7fcb4cbcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:56.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.579+0000 7fcb4cbcf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:56.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.579+0000 7fcb4cbcf640 10 monclient(hunting): picked mon.noname-c con 0x7fcb480b0770 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:56.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.579+0000 7fcb4cbcf640 10 monclient(hunting): picked mon.noname-a con 0x7fcb4814a700 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:56.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.579+0000 7fcb4cbcf640 10 monclient(hunting): picked mon.noname-b con 0x7fcb480ac5a0 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:56.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.579+0000 7fcb4cbcf640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb480aba70 con 0x7fcb4814a700 2024-03-29T11:12:56.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.579+0000 7fcb4cbcf640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb480aecb0 con 0x7fcb480b0770 2024-03-29T11:12:56.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.579+0000 7fcb4cbcf640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb480af150 con 0x7fcb480ac5a0 2024-03-29T11:12:56.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.579+0000 7fcb4cbcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:56.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.579+0000 7fcb4cbcf640 10 monclient(hunting): authenticate will time out at 957.891731s 2024-03-29T11:12:56.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.583+0000 7fcb4659c640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fcb480b0770 0x7fcb48149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:43731/0 (socket says 172.21.15.170:43731) 2024-03-29T11:12:56.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.583+0000 7fcb4659c640 1 -- 172.21.15.170:0/3204348542 learned_addr learned my addr 172.21.15.170:0/3204348542 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:12:56.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 1 -- 172.21.15.170:0/3204348542 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408093751 0 0) 0x7fcb480af150 con 0x7fcb480ac5a0 2024-03-29T11:12:56.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:56.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:56.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:56.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:12:56.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 1 -- 172.21.15.170:0/3204348542 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb30003520 con 0x7fcb480ac5a0 2024-03-29T11:12:56.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 1 -- 172.21.15.170:0/3204348542 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707395494 0 0) 0x7fcb480aecb0 con 0x7fcb480b0770 2024-03-29T11:12:56.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:56.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:56.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:56.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:12:56.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 1 -- 172.21.15.170:0/3204348542 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb480af150 con 0x7fcb480b0770 2024-03-29T11:12:56.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 1 -- 172.21.15.170:0/3204348542 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215801342 0 0) 0x7fcb480aba70 con 0x7fcb4814a700 2024-03-29T11:12:56.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:56.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:56.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:56.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:12:56.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 1 -- 172.21.15.170:0/3204348542 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb480aecb0 con 0x7fcb4814a700 2024-03-29T11:12:56.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 1 -- 172.21.15.170:0/3204348542 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2559742097 0 0) 0x7fcb30003520 con 0x7fcb480ac5a0 2024-03-29T11:12:56.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 1 -- 172.21.15.170:0/3204348542 >> v1:172.21.15.184:6789/0 conn(0x7fcb480b0770 legacy=0x7fcb48149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:56.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.589+0000 7fcb4559a640 1 -- 172.21.15.170:0/3204348542 >> v1:172.21.15.170:6789/0 conn(0x7fcb4814a700 legacy=0x7fcb480abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:56.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.590+0000 7fcb4559a640 10 monclient: _finish_hunting 0 2024-03-29T11:12:56.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.590+0000 7fcb4559a640 1 monclient: found mon.noname-b 2024-03-29T11:12:56.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.590+0000 7fcb4559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:56.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.590+0000 7fcb4559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:12:56.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.590+0000 7fcb4559a640 1 -- 172.21.15.170:0/3204348542 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb480ac9d0 con 0x7fcb480ac5a0 2024-03-29T11:12:56.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.590+0000 7fcb4559a640 10 monclient: _finish_auth 0 2024-03-29T11:12:56.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.590+0000 7fcb4559a640 10 monclient: _check_auth_tickets 2024-03-29T11:12:56.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.590+0000 7fcb4559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:56.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.590+0000 7fcb4cbcf640 5 monclient: authenticate success, global_id 4271 2024-03-29T11:12:56.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.590+0000 7fcb4cbcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:56.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.590+0000 7fcb4559a640 1 -- 172.21.15.170:0/3204348542 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fcb40001a10 con 0x7fcb480ac5a0 2024-03-29T11:12:56.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.590+0000 7fcb4559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:56.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.590+0000 7fcb4559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:56.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.590+0000 7fcb4559a640 10 monclient: dump: 2024-03-29T11:12:56.571 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:56.571 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:56.571 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:56.571 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:56.571 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:56.571 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:56.571 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:56.571 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:56.571 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:56.572 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:56.572 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.590+0000 7fcb4cbcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:56.574 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.590+0000 7fcb4559a640 1 -- 172.21.15.170:0/3204348542 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb40001b70 con 0x7fcb480ac5a0 2024-03-29T11:12:56.574 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.590+0000 7fcb4559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:56.574 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.590+0000 7fcb4cbcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:56.574 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.590+0000 7fcb4cbcf640 4 set_mon_vals no callback set 2024-03-29T11:12:56.574 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.591+0000 7fcb4cbcf640 10 monclient: shutdown 2024-03-29T11:12:56.574 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.591+0000 7fcb4cbcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:56.574 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.591+0000 7fcb4cbcf640 1 -- 172.21.15.170:0/3204348542 >> v1:172.21.15.170:6790/0 conn(0x7fcb480ac5a0 legacy=0x7fcb48151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:56.574 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.592+0000 7fcb4cbcf640 1 -- 172.21.15.170:0/3204348542 shutdown_connections 2024-03-29T11:12:56.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.592+0000 7fcb4cbcf640 1 -- 172.21.15.170:0/3204348542 >> 172.21.15.170:0/3204348542 conn(0x7fcb4806b660 msgr2=0x7fcb4806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:56.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.592+0000 7fcb4cbcf640 1 -- 172.21.15.170:0/3204348542 shutdown_connections 2024-03-29T11:12:56.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.592+0000 7fcb4cbcf640 1 -- 172.21.15.170:0/3204348542 wait complete. 2024-03-29T11:12:56.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.594+0000 7fcb4cbcf640 10 monclient: build_initial_monmap 2024-03-29T11:12:56.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.594+0000 7fcb4cbcf640 10 monclient: monmap: 2024-03-29T11:12:56.575 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:12:56.575 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:56.575 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:12:56.575 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:12:56.575 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:56.575 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:56.575 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:12:56.575 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:12:56.575 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:12:56.576 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:56.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.594+0000 7fcb4cbcf640 1 librados: starting msgr at 2024-03-29T11:12:56.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.594+0000 7fcb4cbcf640 1 librados: starting objecter 2024-03-29T11:12:56.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.594+0000 7fcb4cbcf640 1 Processor -- start 2024-03-29T11:12:56.591 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.595+0000 7fcb4cbcf640 1 -- start start 2024-03-29T11:12:56.591 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.595+0000 7fcb4cbcf640 1 librados: setting wanted keys 2024-03-29T11:12:56.591 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.595+0000 7fcb4cbcf640 1 librados: calling monclient init 2024-03-29T11:12:56.592 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.595+0000 7fcb4cbcf640 10 monclient: init 2024-03-29T11:12:56.592 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.596+0000 7fcb4cbcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:56.592 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.596+0000 7fcb4cbcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:12:56.592 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.596+0000 7fcb4cbcf640 10 monclient(hunting): picked mon.noname-b con 0x7fcb480ac5a0 addr v1:172.21.15.184:6789/0 2024-03-29T11:12:56.592 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.596+0000 7fcb4cbcf640 10 monclient(hunting): picked mon.noname-c con 0x7fcb480b0770 addr v1:172.21.15.170:6790/0 2024-03-29T11:12:56.592 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.596+0000 7fcb4cbcf640 10 monclient(hunting): picked mon.noname-a con 0x7fcb4814a700 addr v1:172.21.15.170:6789/0 2024-03-29T11:12:56.592 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.596+0000 7fcb4cbcf640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4812a960 con 0x7fcb4814a700 2024-03-29T11:12:56.592 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.596+0000 7fcb4cbcf640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb481db960 con 0x7fcb480ac5a0 2024-03-29T11:12:56.592 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.596+0000 7fcb4cbcf640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb481259f0 con 0x7fcb480b0770 2024-03-29T11:12:56.592 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.596+0000 7fcb4cbcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:56.592 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.596+0000 7fcb4cbcf640 10 monclient(hunting): authenticate will time out at 957.908794s 2024-03-29T11:12:56.592 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.600+0000 7fcb4659c640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fcb480ac5a0 0x7fcb481286e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:43859/0 (socket says 172.21.15.170:43859) 2024-03-29T11:12:56.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.600+0000 7fcb4659c640 1 -- 172.21.15.170:0/935940155 learned_addr learned my addr 172.21.15.170:0/935940155 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:12:56.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.602+0000 7fcb36ffd640 10 client.?.objecter ms_handle_connect 0x7fcb480ac5a0 2024-03-29T11:12:56.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.602+0000 7fcb36ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:56.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.602+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270367538 0 0) 0x7fcb481db960 con 0x7fcb480ac5a0 2024-03-29T11:12:56.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.602+0000 7fcb36ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:56.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.602+0000 7fcb36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:56.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.602+0000 7fcb36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:56.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.602+0000 7fcb36ffd640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:12:56.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.602+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb28004eb0 con 0x7fcb480ac5a0 2024-03-29T11:12:56.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.603+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3779976310 0 0) 0x7fcb28004eb0 con 0x7fcb480ac5a0 2024-03-29T11:12:56.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.603+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb481db960 con 0x7fcb480ac5a0 2024-03-29T11:12:56.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.603+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fcb38001970 con 0x7fcb480ac5a0 2024-03-29T11:12:56.594 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.603+0000 7fcb36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:56.594 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.603+0000 7fcb36ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:56.594 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.603+0000 7fcb36ffd640 10 monclient(hunting): dump: 2024-03-29T11:12:56.594 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:56.594 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:56.594 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:56.596 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:56.596 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:56.596 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:56.596 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:56.596 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:56.596 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:56.596 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:56.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.603+0000 7fcb36ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:56.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.604+0000 7fcb36ffd640 10 client.?.objecter ms_handle_connect 0x7fcb480b0770 2024-03-29T11:12:56.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.604+0000 7fcb36ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:56.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.604+0000 7fcb36ffd640 10 client.?.objecter ms_handle_connect 0x7fcb4814a700 2024-03-29T11:12:56.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.604+0000 7fcb36ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:56.597 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.604+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424757752 0 0) 0x7fcb481259f0 con 0x7fcb480b0770 2024-03-29T11:12:56.597 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.604+0000 7fcb36ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:56.597 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.604+0000 7fcb36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:56.597 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.604+0000 7fcb36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:56.597 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.604+0000 7fcb36ffd640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:12:56.597 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.604+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb28004eb0 con 0x7fcb480b0770 2024-03-29T11:12:56.597 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.605+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 336807419 0 0) 0x7fcb28004eb0 con 0x7fcb480b0770 2024-03-29T11:12:56.597 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.605+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb481259f0 con 0x7fcb480b0770 2024-03-29T11:12:56.597 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.605+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fcb3c001810 con 0x7fcb480b0770 2024-03-29T11:12:56.597 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.605+0000 7fcb36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:56.597 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.605+0000 7fcb36ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:56.598 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.605+0000 7fcb36ffd640 10 monclient(hunting): dump: 2024-03-29T11:12:56.598 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:56.598 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:56.598 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:56.598 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:56.598 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:56.598 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:56.598 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:56.598 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:56.598 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:56.598 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:56.598 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.605+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4087509203 0 0) 0x7fcb481db960 con 0x7fcb480ac5a0 2024-03-29T11:12:56.598 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.605+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 >> v1:172.21.15.170:6790/0 conn(0x7fcb480b0770 legacy=0x7fcb48129140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:56.598 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.605+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 >> v1:172.21.15.170:6789/0 conn(0x7fcb4814a700 legacy=0x7fcb481dbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:56.598 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.605+0000 7fcb36ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:56.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.605+0000 7fcb36ffd640 1 monclient: found mon.b 2024-03-29T11:12:56.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.605+0000 7fcb36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:56.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.605+0000 7fcb36ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:56.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.605+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb48125b80 con 0x7fcb480ac5a0 2024-03-29T11:12:56.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.605+0000 7fcb36ffd640 10 monclient: _finish_auth 0 2024-03-29T11:12:56.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.605+0000 7fcb36ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:56.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.605+0000 7fcb36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:56.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.606+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb38001d50 con 0x7fcb480ac5a0 2024-03-29T11:12:56.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.606+0000 7fcb36ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:56.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.606+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fcb38001bd0 con 0x7fcb480ac5a0 2024-03-29T11:12:56.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.606+0000 7fcb36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:56.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.606+0000 7fcb36ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:56.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.606+0000 7fcb44d99640 4 set_mon_vals no callback set 2024-03-29T11:12:56.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.606+0000 7fcb36ffd640 10 monclient: dump: 2024-03-29T11:12:56.600 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:12:56.600 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:12:56.600 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:56.600 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:12:56.600 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:56.600 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:12:56.600 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:12:56.600 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:12:56.600 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:12:56.600 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:12:56.600 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.606+0000 7fcb4cbcf640 5 monclient: authenticate success, global_id 4258 2024-03-29T11:12:56.600 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.606+0000 7fcb4cbcf640 10 monclient: _renew_subs 2024-03-29T11:12:56.600 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.606+0000 7fcb4cbcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:56.600 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.606+0000 7fcb4cbcf640 1 -- 172.21.15.170:0/935940155 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb481dc520 con 0x7fcb480ac5a0 2024-03-29T11:12:56.600 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.606+0000 7fcb4cbcf640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:56.601 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.606+0000 7fcb4cbcf640 10 monclient: _renew_subs 2024-03-29T11:12:56.601 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.606+0000 7fcb4cbcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:56.601 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.606+0000 7fcb4cbcf640 1 -- 172.21.15.170:0/935940155 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb481473e0 con 0x7fcb480ac5a0 2024-03-29T11:12:56.601 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.606+0000 7fcb4cbcf640 1 librados: init done 2024-03-29T11:12:56.601 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.610+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fcb3801a900 con 0x7fcb480ac5a0 2024-03-29T11:12:56.601 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.611+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fcb38084960 con 0x7fcb480ac5a0 2024-03-29T11:12:56.601 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.611+0000 7fcb36ffd640 10 client.4258.objecter ms_dispatch 0x7fcb48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:56.601 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.611+0000 7fcb36ffd640 3 client.4258.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:56.601 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.611+0000 7fcb36ffd640 3 client.4258.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:56.601 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.611+0000 7fcb4cbcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:56.601 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.611+0000 7fcb45d9b640 10 monclient: get_auth_request con 0x7fcb2806ded0 auth_method 0 2024-03-29T11:12:56.601 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.611+0000 7fcb4cbcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:56.601 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.611+0000 7fcb4cbcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:56.601 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.611+0000 7fcb4cbcf640 1 -- 172.21.15.170:0/935940155 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb480ab560 con 0x7fcb480ac5a0 2024-03-29T11:12:56.601 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.612+0000 7fcb36ffd640 20 client.4258.objecter dump_active .. 0 homeless 2024-03-29T11:12:56.601 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.612+0000 7fcb36ffd640 10 client.4258.objecter ms_handle_connect 0x7fcb2806ded0 2024-03-29T11:12:56.602 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.619+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcb38050070 con 0x7fcb480ac5a0 2024-03-29T11:12:56.602 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.619+0000 7fcb36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:56.602 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.619+0000 7fcb36ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:56.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.943+0000 7fcb4cbcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:56.923 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.943+0000 7fcb4cbcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:56.923 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.943+0000 7fcb4cbcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:12:56.924 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.943+0000 7fcb4cbcf640 1 -- 172.21.15.170:0/935940155 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb481dc6b0 con 0x7fcb480ac5a0 2024-03-29T11:12:56.928 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.947+0000 7fcb36ffd640 1 -- 172.21.15.170:0/935940155 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fcb38052170 con 0x7fcb480ac5a0 2024-03-29T11:12:56.928 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.947+0000 7fcb36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:56.928 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.947+0000 7fcb36ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:56.929 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:12:56.929 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:56.929 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.951+0000 7fcb4cbcf640 10 librados: watch_flush enter 2024-03-29T11:12:56.929 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.951+0000 7fcb4cbcf640 10 librados: watch_flush exit 2024-03-29T11:12:56.930 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.951+0000 7fcb4cbcf640 20 client.4258.objecter shutdown clearing up homeless session... 2024-03-29T11:12:56.930 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.951+0000 7fcb4cbcf640 10 client.4258.objecter successfully canceled tick 2024-03-29T11:12:56.930 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.951+0000 7fcb4cbcf640 1 -- 172.21.15.170:0/935940155 >> v1:172.21.15.170:6816/3298994444 conn(0x7fcb2806ded0 legacy=0x7fcb28070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:56.930 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.952+0000 7fcb4cbcf640 10 monclient: shutdown 2024-03-29T11:12:56.930 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.952+0000 7fcb4cbcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:56.930 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.952+0000 7fcb4cbcf640 1 -- 172.21.15.170:0/935940155 >> v1:172.21.15.184:6789/0 conn(0x7fcb480ac5a0 legacy=0x7fcb481286e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:56.930 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.952+0000 7fcb4cbcf640 1 -- 172.21.15.170:0/935940155 shutdown_connections 2024-03-29T11:12:56.930 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.952+0000 7fcb4cbcf640 1 -- 172.21.15.170:0/935940155 >> 172.21.15.170:0/935940155 conn(0x7fcb4806b660 msgr2=0x7fcb4806d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:56.931 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.952+0000 7fcb4cbcf640 1 -- 172.21.15.170:0/935940155 shutdown_connections 2024-03-29T11:12:56.931 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.953+0000 7fcb4cbcf640 1 -- 172.21.15.170:0/935940155 wait complete. 2024-03-29T11:12:56.931 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:12:56.953+0000 7fcb4cbcf640 1 librados: shutdown 2024-03-29T11:12:56.950 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:03.756 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:03.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.932+0000 7f5613a14640 10 monclient: get_monmap_and_config 2024-03-29T11:13:03.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.932+0000 7f5613a14640 10 monclient: build_initial_monmap 2024-03-29T11:13:03.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.932+0000 7f5613a14640 10 monclient: monmap: 2024-03-29T11:13:03.911 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:13:03.911 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:03.911 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:13:03.933779+0000 2024-03-29T11:13:03.911 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:13:03.933779+0000 2024-03-29T11:13:03.911 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:03.911 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:03.911 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:13:03.911 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:13:03.911 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:13:03.912 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:03.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.933+0000 7f5613a14640 1 Processor -- start 2024-03-29T11:13:03.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.933+0000 7f5613a14640 1 -- start start 2024-03-29T11:13:03.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.933+0000 7f5613a14640 10 monclient: init 2024-03-29T11:13:03.913 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.934+0000 7f5613a14640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:03.913 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.934+0000 7f5613a14640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:13:03.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.937+0000 7f5613a14640 10 monclient(hunting): picked mon.noname-a con 0x7f560c0b0770 addr v1:172.21.15.170:6789/0 2024-03-29T11:13:03.918 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.940+0000 7f56117b0640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f560c0b0770 0x7f560c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:40745/0 (socket says 172.21.15.170:40745) 2024-03-29T11:13:03.919 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.941+0000 7f5613a14640 10 monclient(hunting): picked mon.noname-c con 0x7f560c14a700 addr v1:172.21.15.184:6789/0 2024-03-29T11:13:03.920 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.941+0000 7f5613a14640 10 monclient(hunting): picked mon.noname-b con 0x7f560c0ac5a0 addr v1:172.21.15.170:6790/0 2024-03-29T11:13:03.920 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.941+0000 7f5613a14640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c0aba70 con 0x7f560c0b0770 2024-03-29T11:13:03.920 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.941+0000 7f5613a14640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c0aecb0 con 0x7f560c14a700 2024-03-29T11:13:03.920 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.941+0000 7f5613a14640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c0af150 con 0x7f560c0ac5a0 2024-03-29T11:13:03.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.941+0000 7f5613a14640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:03.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.941+0000 7f5613a14640 10 monclient(hunting): authenticate will time out at 965.253330s 2024-03-29T11:13:03.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.941+0000 7f56117b0640 1 -- 172.21.15.170:0/3493396734 learned_addr learned my addr 172.21.15.170:0/3493396734 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:13:03.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.943+0000 7f560bfff640 1 -- 172.21.15.170:0/3493396734 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946080023 0 0) 0x7f560c0aba70 con 0x7f560c0b0770 2024-03-29T11:13:03.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.943+0000 7f560bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:13:03.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.943+0000 7f560bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:03.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.943+0000 7f560bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:03.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.943+0000 7f560bfff640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:13:03.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.943+0000 7f560bfff640 1 -- 172.21.15.170:0/3493396734 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55f4003520 con 0x7f560c0b0770 2024-03-29T11:13:03.928 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.945+0000 7f560bfff640 1 -- 172.21.15.170:0/3493396734 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565892660 0 0) 0x7f560c0af150 con 0x7f560c0ac5a0 2024-03-29T11:13:03.928 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.945+0000 7f560bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:13:03.928 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.945+0000 7f560bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:03.928 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.945+0000 7f560bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:03.928 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.945+0000 7f560bfff640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:13:03.928 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.945+0000 7f560bfff640 1 -- 172.21.15.170:0/3493396734 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f560c0aba70 con 0x7f560c0ac5a0 2024-03-29T11:13:03.928 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 1 -- 172.21.15.170:0/3493396734 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1316055287 0 0) 0x7f560c0aba70 con 0x7f560c0ac5a0 2024-03-29T11:13:03.928 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 1 -- 172.21.15.170:0/3493396734 >> v1:172.21.15.184:6789/0 conn(0x7f560c14a700 legacy=0x7f560c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:03.928 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 1 -- 172.21.15.170:0/3493396734 >> v1:172.21.15.170:6789/0 conn(0x7f560c0b0770 legacy=0x7f560c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:03.928 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:13:03.928 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 1 monclient: found mon.noname-b 2024-03-29T11:13:03.929 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:03.929 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:13:03.929 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 1 -- 172.21.15.170:0/3493396734 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f560c0ac9d0 con 0x7f560c0ac5a0 2024-03-29T11:13:03.929 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 10 monclient: _finish_auth 0 2024-03-29T11:13:03.929 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:13:03.929 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:03.929 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 1 -- 172.21.15.170:0/3493396734 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5604001670 con 0x7f560c0ac5a0 2024-03-29T11:13:03.929 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:03.929 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:03.929 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 10 monclient: dump: 2024-03-29T11:13:03.929 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:03.929 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:03.929 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:03.929 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:03.929 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:03.930 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:03.930 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:03.930 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:03.930 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:03.933 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:03.933 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f5613a14640 5 monclient: authenticate success, global_id 4277 2024-03-29T11:13:03.933 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f5613a14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:03.933 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 1 -- 172.21.15.170:0/3493396734 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56040017d0 con 0x7f560c0ac5a0 2024-03-29T11:13:03.933 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:03.933 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 1 -- 172.21.15.170:0/3493396734 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5604001ba0 con 0x7f560c0ac5a0 2024-03-29T11:13:03.933 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:03.933 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:03.933 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f560bfff640 10 monclient: dump: 2024-03-29T11:13:03.933 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:03.933 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:03.933 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:03.933 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:03.933 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:03.934 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:03.934 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:03.934 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:03.934 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:03.934 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:03.934 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f5613a14640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:03.934 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.946+0000 7f5613a14640 4 set_mon_vals no callback set 2024-03-29T11:13:03.934 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.947+0000 7f5613a14640 10 monclient: shutdown 2024-03-29T11:13:03.934 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.947+0000 7f5613a14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:03.934 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.947+0000 7f5613a14640 1 -- 172.21.15.170:0/3493396734 >> v1:172.21.15.170:6790/0 conn(0x7f560c0ac5a0 legacy=0x7f560c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:03.934 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.947+0000 7f5613a14640 1 -- 172.21.15.170:0/3493396734 shutdown_connections 2024-03-29T11:13:03.935 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.947+0000 7f5613a14640 1 -- 172.21.15.170:0/3493396734 >> 172.21.15.170:0/3493396734 conn(0x7f560c06b660 msgr2=0x7f560c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:03.935 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.947+0000 7f5613a14640 1 -- 172.21.15.170:0/3493396734 shutdown_connections 2024-03-29T11:13:03.935 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.947+0000 7f5613a14640 1 -- 172.21.15.170:0/3493396734 wait complete. 2024-03-29T11:13:03.935 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.948+0000 7f5613a14640 10 monclient: build_initial_monmap 2024-03-29T11:13:03.935 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.948+0000 7f5613a14640 10 monclient: monmap: 2024-03-29T11:13:03.935 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:13:03.935 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:03.935 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:13:03.935 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:13:03.935 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:03.935 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:03.935 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:13:03.935 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:13:03.935 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:13:03.935 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:03.936 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.949+0000 7f5613a14640 1 librados: starting msgr at 2024-03-29T11:13:03.936 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.949+0000 7f5613a14640 1 librados: starting objecter 2024-03-29T11:13:03.936 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.949+0000 7f5613a14640 1 Processor -- start 2024-03-29T11:13:03.936 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.949+0000 7f5613a14640 1 -- start start 2024-03-29T11:13:03.936 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.949+0000 7f5613a14640 1 librados: setting wanted keys 2024-03-29T11:13:03.936 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.949+0000 7f5613a14640 1 librados: calling monclient init 2024-03-29T11:13:03.936 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.949+0000 7f5613a14640 10 monclient: init 2024-03-29T11:13:03.936 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.950+0000 7f5613a14640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:03.936 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.950+0000 7f5613a14640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:13:03.947 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.953+0000 7f5613a14640 10 monclient(hunting): picked mon.noname-c con 0x7f560c0ac5a0 addr v1:172.21.15.170:6790/0 2024-03-29T11:13:03.947 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.954+0000 7f56117b0640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f560c0ac5a0 0x7f560c144c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:47221/0 (socket says 172.21.15.170:47221) 2024-03-29T11:13:03.947 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.954+0000 7f56117b0640 1 -- 172.21.15.170:0/3659965427 learned_addr learned my addr 172.21.15.170:0/3659965427 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:13:03.947 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.954+0000 7f5613a14640 10 monclient(hunting): picked mon.noname-a con 0x7f560c0b0770 addr v1:172.21.15.170:6789/0 2024-03-29T11:13:03.947 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.955+0000 7f5609ffb640 10 client.?.objecter ms_handle_connect 0x7f560c0ac5a0 2024-03-29T11:13:03.947 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.955+0000 7f5609ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:03.947 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.955+0000 7f5613a14640 10 monclient(hunting): picked mon.noname-b con 0x7f560c14a700 addr v1:172.21.15.184:6789/0 2024-03-29T11:13:03.948 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.955+0000 7f5613a14640 1 -- 172.21.15.170:0/3659965427 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c125b60 con 0x7f560c0b0770 2024-03-29T11:13:03.948 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.955+0000 7f5613a14640 1 -- 172.21.15.170:0/3659965427 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c125e40 con 0x7f560c14a700 2024-03-29T11:13:03.948 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.955+0000 7f5613a14640 1 -- 172.21.15.170:0/3659965427 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c145970 con 0x7f560c0ac5a0 2024-03-29T11:13:03.948 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.955+0000 7f5613a14640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:03.948 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.955+0000 7f5613a14640 10 monclient(hunting): authenticate will time out at 965.267526s 2024-03-29T11:13:03.948 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.957+0000 7f5609ffb640 10 client.?.objecter ms_handle_connect 0x7f560c0b0770 2024-03-29T11:13:03.948 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.957+0000 7f5609ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:03.948 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.957+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362573717 0 0) 0x7f560c125b60 con 0x7f560c0b0770 2024-03-29T11:13:03.948 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.957+0000 7f5609ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:13:03.948 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.957+0000 7f5609ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:03.948 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.957+0000 7f5609ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:03.948 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.957+0000 7f5609ffb640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:13:03.948 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.957+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55f8004eb0 con 0x7f560c0b0770 2024-03-29T11:13:03.948 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 10 client.?.objecter ms_handle_connect 0x7f560c14a700 2024-03-29T11:13:03.948 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:03.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172144742 0 0) 0x7f560c145970 con 0x7f560c0ac5a0 2024-03-29T11:13:03.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:13:03.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:03.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:03.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:13:03.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f560c125b60 con 0x7f560c0ac5a0 2024-03-29T11:13:03.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600963967 0 0) 0x7f560c125e40 con 0x7f560c14a700 2024-03-29T11:13:03.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:13:03.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:03.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:03.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:13:03.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f560c145970 con 0x7f560c14a700 2024-03-29T11:13:03.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 227367227 0 0) 0x7f55f8004eb0 con 0x7f560c0b0770 2024-03-29T11:13:03.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f560c125e40 con 0x7f560c0b0770 2024-03-29T11:13:03.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f56000017d0 con 0x7f560c0b0770 2024-03-29T11:13:03.950 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:03.950 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:03.950 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 10 monclient(hunting): dump: 2024-03-29T11:13:03.950 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:03.950 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:03.950 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:03.950 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:03.950 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:03.950 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:03.950 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:03.950 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:03.950 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:03.950 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:03.950 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.958+0000 7f5609ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:03.950 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3670461708 0 0) 0x7f560c145970 con 0x7f560c14a700 2024-03-29T11:13:03.951 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55f8004eb0 con 0x7f560c14a700 2024-03-29T11:13:03.951 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5604001ae0 con 0x7f560c14a700 2024-03-29T11:13:03.951 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5609ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:03.951 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5609ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:03.951 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5609ffb640 10 monclient(hunting): dump: 2024-03-29T11:13:03.951 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:03.951 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:03.951 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:03.951 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:03.951 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:03.951 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:03.951 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:03.951 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:03.951 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:03.951 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:03.952 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2582425602 0 0) 0x7f560c125e40 con 0x7f560c0b0770 2024-03-29T11:13:03.952 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 >> v1:172.21.15.170:6790/0 conn(0x7f560c0ac5a0 legacy=0x7f560c144c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:03.952 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 >> v1:172.21.15.184:6789/0 conn(0x7f560c14a700 legacy=0x7f560c126a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:03.952 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5609ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:13:03.952 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5609ffb640 1 monclient: found mon.a 2024-03-29T11:13:03.952 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5609ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:03.952 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5609ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:13:03.952 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f560c145b00 con 0x7f560c0b0770 2024-03-29T11:13:03.952 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5609ffb640 10 monclient: _finish_auth 0 2024-03-29T11:13:03.952 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5609ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:13:03.952 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:03.952 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5613a14640 5 monclient: authenticate success, global_id 4272 2024-03-29T11:13:03.952 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5613a14640 10 monclient: _renew_subs 2024-03-29T11:13:03.952 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5613a14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:13:03.952 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.959+0000 7f5613a14640 1 -- 172.21.15.170:0/3659965427 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f560c1df470 con 0x7f560c0b0770 2024-03-29T11:13:03.953 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.960+0000 7f5613a14640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:03.953 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.960+0000 7f5613a14640 10 monclient: _renew_subs 2024-03-29T11:13:03.953 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.960+0000 7f5613a14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:13:03.953 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.960+0000 7f5613a14640 1 -- 172.21.15.170:0/3659965427 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f560c1dec10 con 0x7f560c0b0770 2024-03-29T11:13:03.953 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.960+0000 7f5613a14640 1 librados: init done 2024-03-29T11:13:03.953 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.961+0000 7f5613a14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:03.953 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.961+0000 7f5613a14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:03.953 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.961+0000 7f5613a14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:13:03.953 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.961+0000 7f5613a14640 1 -- 172.21.15.170:0/3659965427 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f560c0ab560 con 0x7f560c0b0770 2024-03-29T11:13:03.953 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.963+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5600001ce0 con 0x7f560c0b0770 2024-03-29T11:13:03.953 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.963+0000 7f5609ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:03.953 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.963+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5600003790 con 0x7f560c0b0770 2024-03-29T11:13:03.953 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.963+0000 7f5609ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:03.953 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.963+0000 7f5609ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:03.953 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.963+0000 7f5609ffb640 10 monclient: dump: 2024-03-29T11:13:03.953 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:03.954 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:03.954 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:03.954 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:03.954 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:03.954 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:03.954 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:03.954 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:03.954 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:03.954 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:03.954 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.964+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f56000039b0 con 0x7f560c0b0770 2024-03-29T11:13:03.954 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.964+0000 7f560affd640 4 set_mon_vals no callback set 2024-03-29T11:13:03.954 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.965+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f5600083700 con 0x7f560c0b0770 2024-03-29T11:13:03.954 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.965+0000 7f5609ffb640 10 client.4272.objecter ms_dispatch 0x7f560c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:03.954 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.965+0000 7f5609ffb640 3 client.4272.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:03.954 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.965+0000 7f5609ffb640 3 client.4272.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:03.955 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.965+0000 7f5609ffb640 20 client.4272.objecter dump_active .. 0 homeless 2024-03-29T11:13:03.955 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.965+0000 7f56117b0640 10 monclient: get_auth_request con 0x7f55f806dec0 auth_method 0 2024-03-29T11:13:03.955 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.965+0000 7f5609ffb640 10 client.4272.objecter ms_handle_connect 0x7f55f806dec0 2024-03-29T11:13:03.955 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.971+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f560004efe0 con 0x7f560c0b0770 2024-03-29T11:13:03.955 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.971+0000 7f5609ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:03.955 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:03.971+0000 7f5609ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:04.278 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.300+0000 7f5613a14640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:04.278 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.300+0000 7f5613a14640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:04.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.300+0000 7f5613a14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:13:04.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.300+0000 7f5613a14640 1 -- 172.21.15.170:0/3659965427 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f560c125fd0 con 0x7f560c0b0770 2024-03-29T11:13:04.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.301+0000 7f5609ffb640 1 -- 172.21.15.170:0/3659965427 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f5600050f10 con 0x7f560c0b0770 2024-03-29T11:13:04.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.301+0000 7f5609ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:04.280 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.301+0000 7f5609ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:04.283 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:13:04.283 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:04.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.310+0000 7f5613a14640 10 librados: watch_flush enter 2024-03-29T11:13:04.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.310+0000 7f5613a14640 10 librados: watch_flush exit 2024-03-29T11:13:04.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.310+0000 7f5613a14640 20 client.4272.objecter shutdown clearing up homeless session... 2024-03-29T11:13:04.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.310+0000 7f5613a14640 10 client.4272.objecter successfully canceled tick 2024-03-29T11:13:04.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.310+0000 7f5613a14640 1 -- 172.21.15.170:0/3659965427 >> v1:172.21.15.170:6816/3298994444 conn(0x7f55f806dec0 legacy=0x7f55f8070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:04.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.310+0000 7f5613a14640 10 monclient: shutdown 2024-03-29T11:13:04.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.310+0000 7f5613a14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:04.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.310+0000 7f5613a14640 1 -- 172.21.15.170:0/3659965427 >> v1:172.21.15.170:6789/0 conn(0x7f560c0b0770 legacy=0x7f560c126360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:04.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.310+0000 7f5613a14640 1 -- 172.21.15.170:0/3659965427 shutdown_connections 2024-03-29T11:13:04.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.310+0000 7f5613a14640 1 -- 172.21.15.170:0/3659965427 >> 172.21.15.170:0/3659965427 conn(0x7f560c06b660 msgr2=0x7f560c06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:04.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.310+0000 7f5613a14640 1 -- 172.21.15.170:0/3659965427 shutdown_connections 2024-03-29T11:13:04.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.310+0000 7f5613a14640 1 -- 172.21.15.170:0/3659965427 wait complete. 2024-03-29T11:13:04.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:04.311+0000 7f5613a14640 1 librados: shutdown 2024-03-29T11:13:04.307 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:11.114 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:11.281 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.303+0000 7f91b1a33640 10 monclient: get_monmap_and_config 2024-03-29T11:13:11.281 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.303+0000 7f91b1a33640 10 monclient: build_initial_monmap 2024-03-29T11:13:11.281 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.303+0000 7f91b1a33640 10 monclient: monmap: 2024-03-29T11:13:11.281 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:13:11.281 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:11.282 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:13:11.304620+0000 2024-03-29T11:13:11.282 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:13:11.304620+0000 2024-03-29T11:13:11.282 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:11.282 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:11.282 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:13:11.282 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:13:11.282 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:13:11.282 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:11.286 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.307+0000 7f91b1a33640 1 Processor -- start 2024-03-29T11:13:11.286 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.307+0000 7f91b1a33640 1 -- start start 2024-03-29T11:13:11.286 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.307+0000 7f91b1a33640 10 monclient: init 2024-03-29T11:13:11.287 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.308+0000 7f91b1a33640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:11.287 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.308+0000 7f91b1a33640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:13:11.287 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.308+0000 7f91b1a33640 10 monclient(hunting): picked mon.noname-a con 0x7f91ac0b4330 addr v1:172.21.15.170:6789/0 2024-03-29T11:13:11.287 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.308+0000 7f91b1a33640 10 monclient(hunting): picked mon.noname-b con 0x7f91ac1470b0 addr v1:172.21.15.170:6790/0 2024-03-29T11:13:11.292 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.314+0000 7f91aaffd640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f91ac0b4330 0x7f91ac0b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:45661/0 (socket says 172.21.15.170:45661) 2024-03-29T11:13:11.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.315+0000 7f91b1a33640 10 monclient(hunting): picked mon.noname-c con 0x7f91ac1439b0 addr v1:172.21.15.184:6789/0 2024-03-29T11:13:11.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.315+0000 7f91b1a33640 1 -- 172.21.15.170:0/2138525938 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac06f2a0 con 0x7f91ac0b4330 2024-03-29T11:13:11.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.315+0000 7f91b1a33640 1 -- 172.21.15.170:0/2138525938 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac146b90 con 0x7f91ac1439b0 2024-03-29T11:13:11.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.315+0000 7f91b1a33640 1 -- 172.21.15.170:0/2138525938 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac0b2ca0 con 0x7f91ac1470b0 2024-03-29T11:13:11.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.315+0000 7f91b1a33640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:11.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.315+0000 7f91b1a33640 10 monclient(hunting): authenticate will time out at 972.627471s 2024-03-29T11:13:11.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.315+0000 7f91aaffd640 1 -- 172.21.15.170:0/2138525938 learned_addr learned my addr 172.21.15.170:0/2138525938 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:13:11.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.320+0000 7f91a9ffb640 1 -- 172.21.15.170:0/2138525938 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673783278 0 0) 0x7f91ac146b90 con 0x7f91ac1439b0 2024-03-29T11:13:11.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.320+0000 7f91a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:13:11.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.320+0000 7f91a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:11.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.320+0000 7f91a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:11.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.320+0000 7f91a9ffb640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:13:11.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.320+0000 7f91a9ffb640 1 -- 172.21.15.170:0/2138525938 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9194003520 con 0x7f91ac1439b0 2024-03-29T11:13:11.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.320+0000 7f91a9ffb640 1 -- 172.21.15.170:0/2138525938 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43947546 0 0) 0x7f91ac0b2ca0 con 0x7f91ac1470b0 2024-03-29T11:13:11.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.320+0000 7f91a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:13:11.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.320+0000 7f91a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:11.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.320+0000 7f91a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:11.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.320+0000 7f91a9ffb640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:13:11.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.320+0000 7f91a9ffb640 1 -- 172.21.15.170:0/2138525938 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91ac146b90 con 0x7f91ac1470b0 2024-03-29T11:13:11.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 1 -- 172.21.15.170:0/2138525938 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1623634902 0 0) 0x7f9194003520 con 0x7f91ac1439b0 2024-03-29T11:13:11.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 1 -- 172.21.15.170:0/2138525938 >> v1:172.21.15.170:6790/0 conn(0x7f91ac1470b0 legacy=0x7f91ac0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:11.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 1 -- 172.21.15.170:0/2138525938 >> v1:172.21.15.170:6789/0 conn(0x7f91ac0b4330 legacy=0x7f91ac0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:11.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:13:11.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 1 monclient: found mon.noname-c 2024-03-29T11:13:11.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:11.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:13:11.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 1 -- 172.21.15.170:0/2138525938 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ac06e0c0 con 0x7f91ac1439b0 2024-03-29T11:13:11.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:13:11.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:13:11.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:11.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 1 -- 172.21.15.170:0/2138525938 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f919c001690 con 0x7f91ac1439b0 2024-03-29T11:13:11.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:11.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:11.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 10 monclient: dump: 2024-03-29T11:13:11.303 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:11.303 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:11.303 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:11.303 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:11.303 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:11.304 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:11.304 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:11.304 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:11.304 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:11.304 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:11.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91b1a33640 5 monclient: authenticate success, global_id 4264 2024-03-29T11:13:11.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91b1a33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:11.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 1 -- 172.21.15.170:0/2138525938 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f919c0017f0 con 0x7f91ac1439b0 2024-03-29T11:13:11.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:11.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 1 -- 172.21.15.170:0/2138525938 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f919c001c00 con 0x7f91ac1439b0 2024-03-29T11:13:11.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:11.306 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:11.306 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91a9ffb640 10 monclient: dump: 2024-03-29T11:13:11.306 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:11.306 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:11.306 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:11.306 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:11.306 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:11.306 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:11.306 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:11.307 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:11.307 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:11.307 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:11.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91b1a33640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:11.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.321+0000 7f91b1a33640 4 set_mon_vals no callback set 2024-03-29T11:13:11.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.322+0000 7f91b1a33640 10 monclient: shutdown 2024-03-29T11:13:11.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.322+0000 7f91b1a33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:11.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.322+0000 7f91b1a33640 1 -- 172.21.15.170:0/2138525938 >> v1:172.21.15.184:6789/0 conn(0x7f91ac1439b0 legacy=0x7f91ac151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:11.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.322+0000 7f91b1a33640 1 -- 172.21.15.170:0/2138525938 shutdown_connections 2024-03-29T11:13:11.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.322+0000 7f91b1a33640 1 -- 172.21.15.170:0/2138525938 >> 172.21.15.170:0/2138525938 conn(0x7f91ac066a00 msgr2=0x7f91ac066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:11.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.322+0000 7f91b1a33640 1 -- 172.21.15.170:0/2138525938 shutdown_connections 2024-03-29T11:13:11.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.323+0000 7f91b1a33640 1 -- 172.21.15.170:0/2138525938 wait complete. 2024-03-29T11:13:11.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.323+0000 7f91b1a33640 10 monclient: build_initial_monmap 2024-03-29T11:13:11.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.323+0000 7f91b1a33640 10 monclient: monmap: 2024-03-29T11:13:11.309 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:13:11.309 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:11.309 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:13:11.310 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:13:11.310 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:11.310 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:11.310 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:13:11.310 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:13:11.310 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:13:11.310 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:11.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.324+0000 7f91b1a33640 1 librados: starting msgr at 2024-03-29T11:13:11.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.324+0000 7f91b1a33640 1 librados: starting objecter 2024-03-29T11:13:11.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.324+0000 7f91b1a33640 1 Processor -- start 2024-03-29T11:13:11.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.324+0000 7f91b1a33640 1 -- start start 2024-03-29T11:13:11.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.324+0000 7f91b1a33640 1 librados: setting wanted keys 2024-03-29T11:13:11.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.324+0000 7f91b1a33640 1 librados: calling monclient init 2024-03-29T11:13:11.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.324+0000 7f91b1a33640 10 monclient: init 2024-03-29T11:13:11.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.325+0000 7f91b1a33640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:11.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.325+0000 7f91b1a33640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:13:11.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.329+0000 7f91b1a33640 10 monclient(hunting): picked mon.noname-c con 0x7f91ac0b4330 addr v1:172.21.15.170:6790/0 2024-03-29T11:13:11.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.329+0000 7f91b1a33640 10 monclient(hunting): picked mon.noname-a con 0x7f91ac1439b0 addr v1:172.21.15.170:6789/0 2024-03-29T11:13:11.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.329+0000 7f91aaffd640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f91ac0b4330 0x7f91ac1df030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:53693/0 (socket says 172.21.15.170:53693) 2024-03-29T11:13:11.312 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.333+0000 7f91aaffd640 1 -- 172.21.15.170:0/1871089018 learned_addr learned my addr 172.21.15.170:0/1871089018 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:13:11.312 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.333+0000 7f91b1a33640 10 monclient(hunting): picked mon.noname-b con 0x7f91ac1470b0 addr v1:172.21.15.184:6789/0 2024-03-29T11:13:11.312 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.333+0000 7f91b1a33640 1 -- 172.21.15.170:0/1871089018 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac13f7a0 con 0x7f91ac1439b0 2024-03-29T11:13:11.312 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.333+0000 7f91b1a33640 1 -- 172.21.15.170:0/1871089018 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac1df740 con 0x7f91ac1470b0 2024-03-29T11:13:11.312 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.333+0000 7f91b1a33640 1 -- 172.21.15.170:0/1871089018 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac1df8d0 con 0x7f91ac0b4330 2024-03-29T11:13:11.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.333+0000 7f91b1a33640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:11.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.333+0000 7f91b1a33640 10 monclient(hunting): authenticate will time out at 972.645744s 2024-03-29T11:13:11.314 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.336+0000 7f9193fff640 10 client.?.objecter ms_handle_connect 0x7f91ac0b4330 2024-03-29T11:13:11.314 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.336+0000 7f9193fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:11.315 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.336+0000 7f9193fff640 10 client.?.objecter ms_handle_connect 0x7f91ac1470b0 2024-03-29T11:13:11.315 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.336+0000 7f9193fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:11.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 10 client.?.objecter ms_handle_connect 0x7f91ac1439b0 2024-03-29T11:13:11.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:11.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869461147 0 0) 0x7f91ac1df740 con 0x7f91ac1470b0 2024-03-29T11:13:11.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 10 monclient(hunting): _negotiate 2024-03-29T11:13:11.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:11.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:11.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:13:11.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9198004eb0 con 0x7f91ac1470b0 2024-03-29T11:13:11.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773756346 0 0) 0x7f91ac13f7a0 con 0x7f91ac1439b0 2024-03-29T11:13:11.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 10 monclient(hunting): _negotiate 2024-03-29T11:13:11.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:11.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:11.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:13:11.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91ac1df740 con 0x7f91ac1439b0 2024-03-29T11:13:11.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603641644 0 0) 0x7f91ac1df8d0 con 0x7f91ac0b4330 2024-03-29T11:13:11.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 10 monclient(hunting): _negotiate 2024-03-29T11:13:11.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:11.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:11.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:13:11.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91ac13f7a0 con 0x7f91ac0b4330 2024-03-29T11:13:11.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 698400669 0 0) 0x7f9198004eb0 con 0x7f91ac1470b0 2024-03-29T11:13:11.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91ac1df8d0 con 0x7f91ac1470b0 2024-03-29T11:13:11.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f919c001bc0 con 0x7f91ac1470b0 2024-03-29T11:13:11.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:11.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:11.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 10 monclient(hunting): dump: 2024-03-29T11:13:11.317 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:11.317 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:11.317 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:11.317 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:11.318 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:11.318 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:11.318 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:11.318 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:11.318 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:11.318 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:11.318 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.337+0000 7f9193fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:11.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 266382029 0 0) 0x7f91ac13f7a0 con 0x7f91ac0b4330 2024-03-29T11:13:11.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9198004eb0 con 0x7f91ac0b4330 2024-03-29T11:13:11.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f91a4001a80 con 0x7f91ac0b4330 2024-03-29T11:13:11.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f9193fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:11.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f9193fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:11.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f9193fff640 10 monclient(hunting): dump: 2024-03-29T11:13:11.324 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:11.324 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:11.324 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:11.324 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:11.324 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:11.324 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:11.324 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:11.324 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:11.324 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:11.325 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:11.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4067491513 0 0) 0x7f91ac1df8d0 con 0x7f91ac1470b0 2024-03-29T11:13:11.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 >> v1:172.21.15.170:6790/0 conn(0x7f91ac0b4330 legacy=0x7f91ac1df030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:11.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 >> v1:172.21.15.170:6789/0 conn(0x7f91ac1439b0 legacy=0x7f91ac123030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:11.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f9193fff640 10 monclient: _finish_hunting 0 2024-03-29T11:13:11.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f9193fff640 1 monclient: found mon.b 2024-03-29T11:13:11.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f9193fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:11.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f9193fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:13:11.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ac13ec90 con 0x7f91ac1470b0 2024-03-29T11:13:11.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f9193fff640 10 monclient: _finish_auth 0 2024-03-29T11:13:11.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f9193fff640 10 monclient: _check_auth_tickets 2024-03-29T11:13:11.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f9193fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:11.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f91b1a33640 5 monclient: authenticate success, global_id 4267 2024-03-29T11:13:11.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f91b1a33640 10 monclient: _renew_subs 2024-03-29T11:13:11.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f91b1a33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:13:11.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f91b1a33640 1 -- 172.21.15.170:0/1871089018 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ac13eee0 con 0x7f91ac1470b0 2024-03-29T11:13:11.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f91b1a33640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:11.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f91b1a33640 10 monclient: _renew_subs 2024-03-29T11:13:11.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f91b1a33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:13:11.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f91b1a33640 1 -- 172.21.15.170:0/1871089018 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ac1e7560 con 0x7f91ac1470b0 2024-03-29T11:13:11.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.340+0000 7f91b1a33640 1 librados: init done 2024-03-29T11:13:11.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.343+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f919c003910 con 0x7f91ac1470b0 2024-03-29T11:13:11.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.343+0000 7f9193fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:11.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.343+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f919c003c60 con 0x7f91ac1470b0 2024-03-29T11:13:11.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.343+0000 7f9193fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:11.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.343+0000 7f9193fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:11.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.343+0000 7f9193fff640 10 monclient: dump: 2024-03-29T11:13:11.326 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:11.327 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:11.327 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:11.327 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:11.327 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:11.327 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:11.327 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:11.327 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:11.327 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:11.327 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:11.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.343+0000 7f91a97fa640 4 set_mon_vals no callback set 2024-03-29T11:13:11.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.344+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f919c003e80 con 0x7f91ac1470b0 2024-03-29T11:13:11.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.344+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f919c083930 con 0x7f91ac1470b0 2024-03-29T11:13:11.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.345+0000 7f9193fff640 10 client.4267.objecter ms_dispatch 0x7f91ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:11.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.345+0000 7f9193fff640 3 client.4267.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:11.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.345+0000 7f9193fff640 3 client.4267.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:11.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.345+0000 7f91b1a33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:11.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.345+0000 7f9193fff640 20 client.4267.objecter dump_active .. 0 homeless 2024-03-29T11:13:11.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.345+0000 7f91b1a33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:11.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.345+0000 7f91b1a33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:13:11.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.345+0000 7f91b1a33640 1 -- 172.21.15.170:0/1871089018 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918c005080 con 0x7f91ac1470b0 2024-03-29T11:13:11.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.346+0000 7f91aaffd640 10 monclient: get_auth_request con 0x7f919806dfe0 auth_method 0 2024-03-29T11:13:11.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.350+0000 7f9193fff640 10 client.4267.objecter ms_handle_connect 0x7f919806dfe0 2024-03-29T11:13:11.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.351+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f919c04f210 con 0x7f91ac1470b0 2024-03-29T11:13:11.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.351+0000 7f9193fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:11.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.351+0000 7f9193fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:11.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.751+0000 7f91b1a33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:11.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.751+0000 7f91b1a33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:11.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.751+0000 7f91b1a33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:13:11.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.751+0000 7f91b1a33640 1 -- 172.21.15.170:0/1871089018 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f918c005330 con 0x7f91ac1470b0 2024-03-29T11:13:11.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.752+0000 7f9193fff640 1 -- 172.21.15.170:0/1871089018 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f919c051140 con 0x7f91ac1470b0 2024-03-29T11:13:11.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.752+0000 7f9193fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:11.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.752+0000 7f9193fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:11.740 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:13:11.740 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:11.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.756+0000 7f91b1a33640 10 librados: watch_flush enter 2024-03-29T11:13:11.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.756+0000 7f91b1a33640 10 librados: watch_flush exit 2024-03-29T11:13:11.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.756+0000 7f91b1a33640 20 client.4267.objecter shutdown clearing up homeless session... 2024-03-29T11:13:11.742 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.756+0000 7f91b1a33640 10 client.4267.objecter successfully canceled tick 2024-03-29T11:13:11.742 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.757+0000 7f91b1a33640 1 -- 172.21.15.170:0/1871089018 >> v1:172.21.15.170:6816/3298994444 conn(0x7f919806dfe0 legacy=0x7f91980704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:11.742 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.757+0000 7f91b1a33640 10 monclient: shutdown 2024-03-29T11:13:11.742 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.757+0000 7f91b1a33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:11.742 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.757+0000 7f91b1a33640 1 -- 172.21.15.170:0/1871089018 >> v1:172.21.15.184:6789/0 conn(0x7f91ac1470b0 legacy=0x7f91ac123740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:11.742 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.757+0000 7f91b1a33640 1 -- 172.21.15.170:0/1871089018 shutdown_connections 2024-03-29T11:13:11.742 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.757+0000 7f91b1a33640 1 -- 172.21.15.170:0/1871089018 >> 172.21.15.170:0/1871089018 conn(0x7f91ac066a00 msgr2=0x7f91ac070f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:11.742 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.757+0000 7f91b1a33640 1 -- 172.21.15.170:0/1871089018 shutdown_connections 2024-03-29T11:13:11.742 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.757+0000 7f91b1a33640 1 -- 172.21.15.170:0/1871089018 wait complete. 2024-03-29T11:13:11.742 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:11.757+0000 7f91b1a33640 1 librados: shutdown 2024-03-29T11:13:11.752 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:18.560 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:18.716 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.739+0000 7fa711146640 10 monclient: get_monmap_and_config 2024-03-29T11:13:18.716 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.739+0000 7fa711146640 10 monclient: build_initial_monmap 2024-03-29T11:13:18.716 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.739+0000 7fa711146640 10 monclient: monmap: 2024-03-29T11:13:18.717 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:13:18.717 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:18.717 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:13:18.739867+0000 2024-03-29T11:13:18.717 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:13:18.739867+0000 2024-03-29T11:13:18.717 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:18.717 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:18.717 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:13:18.717 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:13:18.717 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:13:18.717 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:18.718 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.740+0000 7fa711146640 1 Processor -- start 2024-03-29T11:13:18.718 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.740+0000 7fa711146640 1 -- start start 2024-03-29T11:13:18.718 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.740+0000 7fa711146640 10 monclient: init 2024-03-29T11:13:18.719 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.741+0000 7fa711146640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:18.719 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.741+0000 7fa711146640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:13:18.719 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.741+0000 7fa711146640 10 monclient(hunting): picked mon.noname-b con 0x7fa704038c60 addr v1:172.21.15.170:6790/0 2024-03-29T11:13:18.719 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.741+0000 7fa711146640 10 monclient(hunting): picked mon.noname-a con 0x7fa7040324b0 addr v1:172.21.15.170:6789/0 2024-03-29T11:13:18.719 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.741+0000 7fa711146640 10 monclient(hunting): picked mon.noname-c con 0x7fa7040d11e0 addr v1:172.21.15.184:6789/0 2024-03-29T11:13:18.719 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.742+0000 7fa711146640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7040caf80 con 0x7fa7040324b0 2024-03-29T11:13:18.720 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.742+0000 7fa711146640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa704039850 con 0x7fa7040d11e0 2024-03-29T11:13:18.720 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.742+0000 7fa711146640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7040c78d0 con 0x7fa704038c60 2024-03-29T11:13:18.720 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.742+0000 7fa711146640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:18.720 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.742+0000 7fa711146640 10 monclient(hunting): authenticate will time out at 980.053045s 2024-03-29T11:13:18.720 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.742+0000 7fa70b7fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fa7040324b0 0x7fa7040cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:41778/0 (socket says 172.21.15.170:41778) 2024-03-29T11:13:18.720 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.742+0000 7fa70b7fe640 1 -- 172.21.15.170:0/2869400581 learned_addr learned my addr 172.21.15.170:0/2869400581 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:13:18.721 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 1 -- 172.21.15.170:0/2869400581 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550151511 0 0) 0x7fa704039850 con 0x7fa7040d11e0 2024-03-29T11:13:18.721 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:18.721 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:18.722 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:18.722 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:13:18.722 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 1 -- 172.21.15.170:0/2869400581 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6f4003520 con 0x7fa7040d11e0 2024-03-29T11:13:18.722 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 1 -- 172.21.15.170:0/2869400581 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3768070774 0 0) 0x7fa6f4003520 con 0x7fa7040d11e0 2024-03-29T11:13:18.722 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 1 -- 172.21.15.170:0/2869400581 >> v1:172.21.15.170:6790/0 conn(0x7fa704038c60 legacy=0x7fa7040c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:18.722 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 1 -- 172.21.15.170:0/2869400581 >> v1:172.21.15.170:6789/0 conn(0x7fa7040324b0 legacy=0x7fa7040cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:18.722 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 10 monclient: _finish_hunting 0 2024-03-29T11:13:18.723 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 1 monclient: found mon.noname-c 2024-03-29T11:13:18.723 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:18.723 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:13:18.723 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 1 -- 172.21.15.170:0/2869400581 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7040c7a60 con 0x7fa7040d11e0 2024-03-29T11:13:18.723 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 10 monclient: _finish_auth 0 2024-03-29T11:13:18.723 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 10 monclient: _check_auth_tickets 2024-03-29T11:13:18.723 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:18.723 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa711146640 5 monclient: authenticate success, global_id 4270 2024-03-29T11:13:18.723 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa711146640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:18.723 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 1 -- 172.21.15.170:0/2869400581 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa70c04f730 con 0x7fa7040d11e0 2024-03-29T11:13:18.723 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:18.723 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:18.723 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa70affd640 10 monclient: dump: 2024-03-29T11:13:18.723 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:18.723 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:18.724 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:18.724 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:18.724 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:18.724 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:18.724 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:18.724 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:18.724 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:18.724 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:18.724 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.743+0000 7fa711146640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:18.724 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.744+0000 7fa70affd640 1 -- 172.21.15.170:0/2869400581 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa70c075f10 con 0x7fa7040d11e0 2024-03-29T11:13:18.724 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.744+0000 7fa70affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:18.724 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.744+0000 7fa70affd640 1 -- 172.21.15.170:0/2869400581 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa70c0758b0 con 0x7fa7040d11e0 2024-03-29T11:13:18.724 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.744+0000 7fa70affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:18.725 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.744+0000 7fa70affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:18.725 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.744+0000 7fa70affd640 10 monclient: dump: 2024-03-29T11:13:18.725 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:18.725 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:18.725 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:18.725 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:18.725 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:18.725 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:18.725 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:18.725 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:18.725 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:18.725 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:18.725 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.744+0000 7fa711146640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:18.725 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.744+0000 7fa711146640 4 set_mon_vals no callback set 2024-03-29T11:13:18.727 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.745+0000 7fa711146640 10 monclient: shutdown 2024-03-29T11:13:18.727 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.745+0000 7fa711146640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:18.727 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.745+0000 7fa711146640 1 -- 172.21.15.170:0/2869400581 >> v1:172.21.15.184:6789/0 conn(0x7fa7040d11e0 legacy=0x7fa7040d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:18.727 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.745+0000 7fa711146640 1 -- 172.21.15.170:0/2869400581 shutdown_connections 2024-03-29T11:13:18.727 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.745+0000 7fa711146640 1 -- 172.21.15.170:0/2869400581 >> 172.21.15.170:0/2869400581 conn(0x7fa704005c20 msgr2=0x7fa704006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:18.727 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.745+0000 7fa711146640 1 -- 172.21.15.170:0/2869400581 shutdown_connections 2024-03-29T11:13:18.728 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.745+0000 7fa711146640 1 -- 172.21.15.170:0/2869400581 wait complete. 2024-03-29T11:13:18.728 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.746+0000 7fa711146640 10 monclient: build_initial_monmap 2024-03-29T11:13:18.728 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.746+0000 7fa711146640 10 monclient: monmap: 2024-03-29T11:13:18.728 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:13:18.728 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:18.728 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:13:18.728 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:13:18.728 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:18.728 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:18.728 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:13:18.729 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:13:18.729 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:13:18.729 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:18.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.747+0000 7fa711146640 1 librados: starting msgr at 2024-03-29T11:13:18.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.747+0000 7fa711146640 1 librados: starting objecter 2024-03-29T11:13:18.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.748+0000 7fa711146640 1 Processor -- start 2024-03-29T11:13:18.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.748+0000 7fa711146640 1 -- start start 2024-03-29T11:13:18.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.748+0000 7fa711146640 1 librados: setting wanted keys 2024-03-29T11:13:18.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.748+0000 7fa711146640 1 librados: calling monclient init 2024-03-29T11:13:18.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.748+0000 7fa711146640 10 monclient: init 2024-03-29T11:13:18.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.749+0000 7fa711146640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:18.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.749+0000 7fa711146640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:13:18.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.749+0000 7fa711146640 10 monclient(hunting): picked mon.noname-c con 0x7fa7040324b0 addr v1:172.21.15.170:6790/0 2024-03-29T11:13:18.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.749+0000 7fa711146640 10 monclient(hunting): picked mon.noname-a con 0x7fa704038c60 addr v1:172.21.15.170:6789/0 2024-03-29T11:13:18.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.749+0000 7fa711146640 10 monclient(hunting): picked mon.noname-b con 0x7fa7040d11e0 addr v1:172.21.15.184:6789/0 2024-03-29T11:13:18.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.749+0000 7fa711146640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa704036ca0 con 0x7fa704038c60 2024-03-29T11:13:18.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.749+0000 7fa711146640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7040370c0 con 0x7fa7040d11e0 2024-03-29T11:13:18.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.749+0000 7fa711146640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa704037250 con 0x7fa7040324b0 2024-03-29T11:13:18.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.749+0000 7fa711146640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:18.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.749+0000 7fa711146640 10 monclient(hunting): authenticate will time out at 980.060946s 2024-03-29T11:13:18.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.750+0000 7fa70b7fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fa704038c60 0x7fa7040be180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:42106/0 (socket says 172.21.15.170:42106) 2024-03-29T11:13:18.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.750+0000 7fa70b7fe640 1 -- 172.21.15.170:0/2007137072 learned_addr learned my addr 172.21.15.170:0/2007137072 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:13:18.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.750+0000 7fa70bfff640 1 --1- 172.21.15.170:0/2007137072 >> v1:172.21.15.170:6790/0 conn(0x7fa7040324b0 0x7fa7041601f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:34820/0 (socket says 172.21.15.170:34820) 2024-03-29T11:13:18.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.750+0000 7fa708ff9640 10 client.?.objecter ms_handle_connect 0x7fa7040324b0 2024-03-29T11:13:18.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.750+0000 7fa708ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:18.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.750+0000 7fa708ff9640 10 client.?.objecter ms_handle_connect 0x7fa7040d11e0 2024-03-29T11:13:18.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.750+0000 7fa708ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:18.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.750+0000 7fa708ff9640 10 client.?.objecter ms_handle_connect 0x7fa704038c60 2024-03-29T11:13:18.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.750+0000 7fa708ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:18.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707287228 0 0) 0x7fa7040370c0 con 0x7fa7040d11e0 2024-03-29T11:13:18.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:13:18.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:18.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:18.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:13:18.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6f8004eb0 con 0x7fa7040d11e0 2024-03-29T11:13:18.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967167514 0 0) 0x7fa704037250 con 0x7fa7040324b0 2024-03-29T11:13:18.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:13:18.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:18.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:18.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:13:18.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7040370c0 con 0x7fa7040324b0 2024-03-29T11:13:18.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538257011 0 0) 0x7fa704036ca0 con 0x7fa704038c60 2024-03-29T11:13:18.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:13:18.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:18.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:18.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:13:18.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa704037250 con 0x7fa704038c60 2024-03-29T11:13:18.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2405403935 0 0) 0x7fa7040370c0 con 0x7fa7040324b0 2024-03-29T11:13:18.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa704036ca0 con 0x7fa7040324b0 2024-03-29T11:13:18.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa7000018e0 con 0x7fa7040324b0 2024-03-29T11:13:18.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:18.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:18.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 10 monclient(hunting): dump: 2024-03-29T11:13:18.736 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:18.736 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:18.736 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:18.736 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:18.736 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:18.736 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:18.736 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:18.736 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:18.736 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:18.736 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:18.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.751+0000 7fa708ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:18.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 384196368 0 0) 0x7fa6f8004eb0 con 0x7fa7040d11e0 2024-03-29T11:13:18.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7040370c0 con 0x7fa7040d11e0 2024-03-29T11:13:18.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa70c04f400 con 0x7fa7040d11e0 2024-03-29T11:13:18.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa708ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:18.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa708ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:18.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa708ff9640 10 monclient(hunting): dump: 2024-03-29T11:13:18.737 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:18.737 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:18.737 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:18.737 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:18.737 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:18.737 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:18.737 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:18.737 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:18.737 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:18.738 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:18.738 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3594786099 0 0) 0x7fa704036ca0 con 0x7fa7040324b0 2024-03-29T11:13:18.738 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 >> v1:172.21.15.184:6789/0 conn(0x7fa7040d11e0 legacy=0x7fa704034990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:18.738 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 >> v1:172.21.15.170:6789/0 conn(0x7fa704038c60 legacy=0x7fa7040be180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:18.738 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa708ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:13:18.738 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa708ff9640 1 monclient: found mon.c 2024-03-29T11:13:18.738 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa708ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:18.738 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa708ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:18.738 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa704160900 con 0x7fa7040324b0 2024-03-29T11:13:18.738 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa708ff9640 10 monclient: _finish_auth 0 2024-03-29T11:13:18.738 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa708ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:13:18.738 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa708ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:18.738 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa711146640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:13:18.738 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa711146640 10 monclient: _renew_subs 2024-03-29T11:13:18.738 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa711146640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:18.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa711146640 1 -- 172.21.15.170:0/2007137072 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa704160bb0 con 0x7fa7040324b0 2024-03-29T11:13:18.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa711146640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:18.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa711146640 10 monclient: _renew_subs 2024-03-29T11:13:18.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa711146640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:18.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa711146640 1 -- 172.21.15.170:0/2007137072 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7040bea70 con 0x7fa7040324b0 2024-03-29T11:13:18.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.752+0000 7fa711146640 1 librados: init done 2024-03-29T11:13:18.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.753+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa700001a40 con 0x7fa7040324b0 2024-03-29T11:13:18.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.753+0000 7fa708ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:18.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.753+0000 7fa70a7fc640 4 set_mon_vals no callback set 2024-03-29T11:13:18.740 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.753+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa7000037b0 con 0x7fa7040324b0 2024-03-29T11:13:18.740 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.753+0000 7fa708ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:18.740 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.753+0000 7fa708ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:18.740 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.753+0000 7fa708ff9640 10 monclient: dump: 2024-03-29T11:13:18.740 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:18.740 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:18.740 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:18.740 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:18.740 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:18.740 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:18.740 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:18.740 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:18.740 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:18.740 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:18.740 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.753+0000 7fa711146640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:18.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.755+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fa700003910 con 0x7fa7040324b0 2024-03-29T11:13:18.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.755+0000 7fa711146640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:18.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.755+0000 7fa711146640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:18.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.755+0000 7fa711146640 1 -- 172.21.15.170:0/2007137072 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ec005080 con 0x7fa7040324b0 2024-03-29T11:13:18.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.756+0000 7fa70b7fe640 10 monclient: get_auth_request con 0x7fa6f806dcc0 auth_method 0 2024-03-29T11:13:18.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.756+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fa700084970 con 0x7fa7040324b0 2024-03-29T11:13:18.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.756+0000 7fa708ff9640 10 client.4292.objecter ms_dispatch 0x7fa70c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:18.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.756+0000 7fa708ff9640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:18.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.756+0000 7fa708ff9640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:18.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.756+0000 7fa708ff9640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-29T11:13:18.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.756+0000 7fa708ff9640 10 client.4292.objecter ms_handle_connect 0x7fa6f806dcc0 2024-03-29T11:13:18.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.766+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa700050250 con 0x7fa7040324b0 2024-03-29T11:13:18.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.766+0000 7fa708ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:18.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:18.766+0000 7fa708ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:19.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.164+0000 7fa711146640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:19.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.164+0000 7fa711146640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:19.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.164+0000 7fa711146640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:19.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.164+0000 7fa711146640 1 -- 172.21.15.170:0/2007137072 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6ec005330 con 0x7fa7040324b0 2024-03-29T11:13:19.142 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.165+0000 7fa708ff9640 1 -- 172.21.15.170:0/2007137072 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fa700052180 con 0x7fa7040324b0 2024-03-29T11:13:19.142 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.165+0000 7fa708ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:19.142 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.165+0000 7fa708ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:19.142 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:13:19.142 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:19.146 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.169+0000 7fa711146640 10 librados: watch_flush enter 2024-03-29T11:13:19.146 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.169+0000 7fa711146640 10 librados: watch_flush exit 2024-03-29T11:13:19.146 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.169+0000 7fa711146640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-29T11:13:19.146 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.169+0000 7fa711146640 10 client.4292.objecter successfully canceled tick 2024-03-29T11:13:19.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.169+0000 7fa711146640 1 -- 172.21.15.170:0/2007137072 >> v1:172.21.15.170:6816/3298994444 conn(0x7fa6f806dcc0 legacy=0x7fa6f8070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:19.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.169+0000 7fa711146640 10 monclient: shutdown 2024-03-29T11:13:19.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.169+0000 7fa711146640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:19.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.169+0000 7fa711146640 1 -- 172.21.15.170:0/2007137072 >> v1:172.21.15.170:6790/0 conn(0x7fa7040324b0 legacy=0x7fa7041601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:19.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.169+0000 7fa711146640 1 -- 172.21.15.170:0/2007137072 shutdown_connections 2024-03-29T11:13:19.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.169+0000 7fa711146640 1 -- 172.21.15.170:0/2007137072 >> 172.21.15.170:0/2007137072 conn(0x7fa704005c20 msgr2=0x7fa704006000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:19.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.170+0000 7fa711146640 1 -- 172.21.15.170:0/2007137072 shutdown_connections 2024-03-29T11:13:19.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.170+0000 7fa711146640 1 -- 172.21.15.170:0/2007137072 wait complete. 2024-03-29T11:13:19.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:19.170+0000 7fa711146640 1 librados: shutdown 2024-03-29T11:13:19.165 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:25.971 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:26.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.132+0000 7f152a6f1640 10 monclient: get_monmap_and_config 2024-03-29T11:13:26.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.132+0000 7f152a6f1640 10 monclient: build_initial_monmap 2024-03-29T11:13:26.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.132+0000 7f152a6f1640 10 monclient: monmap: 2024-03-29T11:13:26.110 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:13:26.110 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:26.111 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:13:26.133361+0000 2024-03-29T11:13:26.111 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:13:26.133361+0000 2024-03-29T11:13:26.111 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:26.111 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:26.111 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:13:26.111 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:13:26.111 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:13:26.111 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:26.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.134+0000 7f152a6f1640 1 Processor -- start 2024-03-29T11:13:26.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.134+0000 7f152a6f1640 1 -- start start 2024-03-29T11:13:26.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.134+0000 7f152a6f1640 10 monclient: init 2024-03-29T11:13:26.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.134+0000 7f152a6f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:26.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.134+0000 7f152a6f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:13:26.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.134+0000 7f152a6f1640 10 monclient(hunting): picked mon.noname-b con 0x7f15241457f0 addr v1:172.21.15.170:6790/0 2024-03-29T11:13:26.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.134+0000 7f152a6f1640 10 monclient(hunting): picked mon.noname-c con 0x7f152413f650 addr v1:172.21.15.184:6789/0 2024-03-29T11:13:26.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.134+0000 7f152a6f1640 10 monclient(hunting): picked mon.noname-a con 0x7f15240a9d70 addr v1:172.21.15.170:6789/0 2024-03-29T11:13:26.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.134+0000 7f152a6f1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15240aaef0 con 0x7f15240a9d70 2024-03-29T11:13:26.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.134+0000 7f152a6f1640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1524145bd0 con 0x7f152413f650 2024-03-29T11:13:26.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.134+0000 7f152a6f1640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1524145eb0 con 0x7f15241457f0 2024-03-29T11:13:26.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.134+0000 7f152a6f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:26.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.134+0000 7f152a6f1640 10 monclient(hunting): authenticate will time out at 987.445964s 2024-03-29T11:13:26.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.135+0000 7f1523fff640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f15241457f0 0x7f152413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:46564/0 (socket says 172.21.15.170:46564) 2024-03-29T11:13:26.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.135+0000 7f1523fff640 1 -- 172.21.15.170:0/1269200896 learned_addr learned my addr 172.21.15.170:0/1269200896 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:13:26.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.135+0000 7f1522ffd640 1 -- 172.21.15.170:0/1269200896 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814987416 0 0) 0x7f1524145eb0 con 0x7f15241457f0 2024-03-29T11:13:26.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.135+0000 7f1522ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:26.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.135+0000 7f1522ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:26.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.135+0000 7f1522ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:26.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.135+0000 7f1522ffd640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:13:26.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.135+0000 7f1522ffd640 1 -- 172.21.15.170:0/1269200896 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f150c003520 con 0x7f15241457f0 2024-03-29T11:13:26.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.135+0000 7f1522ffd640 1 -- 172.21.15.170:0/1269200896 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1953487470 0 0) 0x7f150c003520 con 0x7f15241457f0 2024-03-29T11:13:26.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.135+0000 7f1522ffd640 1 -- 172.21.15.170:0/1269200896 >> v1:172.21.15.184:6789/0 conn(0x7f152413f650 legacy=0x7f15240b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:26.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.135+0000 7f1522ffd640 1 -- 172.21.15.170:0/1269200896 >> v1:172.21.15.170:6789/0 conn(0x7f15240a9d70 legacy=0x7f152414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:13:26.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f1522ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:13:26.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f1522ffd640 1 monclient: found mon.noname-b 2024-03-29T11:13:26.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f1522ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:26.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f1522ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:13:26.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f1522ffd640 1 -- 172.21.15.170:0/1269200896 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1524146340 con 0x7f15241457f0 2024-03-29T11:13:26.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f1522ffd640 10 monclient: _finish_auth 0 2024-03-29T11:13:26.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f1522ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:13:26.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f1522ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:26.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f1522ffd640 1 -- 172.21.15.170:0/1269200896 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f1518001690 con 0x7f15241457f0 2024-03-29T11:13:26.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f1522ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:26.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f1522ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:26.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f1522ffd640 10 monclient: dump: 2024-03-29T11:13:26.135 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:26.135 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:26.135 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:26.135 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:26.135 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:26.135 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:26.136 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:26.136 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:26.136 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:26.136 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:26.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f152a6f1640 5 monclient: authenticate success, global_id 4295 2024-03-29T11:13:26.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f152a6f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:26.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f1522ffd640 1 -- 172.21.15.170:0/1269200896 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1518001860 con 0x7f15241457f0 2024-03-29T11:13:26.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f1522ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:26.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f1522ffd640 1 -- 172.21.15.170:0/1269200896 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f1518001cb0 con 0x7f15241457f0 2024-03-29T11:13:26.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f1522ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:26.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f1522ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:26.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f1522ffd640 10 monclient: dump: 2024-03-29T11:13:26.136 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:26.136 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:26.136 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:26.137 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:26.137 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:26.137 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:26.137 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:26.137 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:26.137 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:26.137 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:26.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f152a6f1640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:26.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.137+0000 7f152a6f1640 4 set_mon_vals no callback set 2024-03-29T11:13:26.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.138+0000 7f152a6f1640 10 monclient: shutdown 2024-03-29T11:13:26.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.138+0000 7f152a6f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:26.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.138+0000 7f152a6f1640 1 -- 172.21.15.170:0/1269200896 >> v1:172.21.15.170:6790/0 conn(0x7f15241457f0 legacy=0x7f152413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:26.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.138+0000 7f152a6f1640 1 -- 172.21.15.170:0/1269200896 shutdown_connections 2024-03-29T11:13:26.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.138+0000 7f152a6f1640 1 -- 172.21.15.170:0/1269200896 >> 172.21.15.170:0/1269200896 conn(0x7f1524066a00 msgr2=0x7f1524066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:26.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.138+0000 7f152a6f1640 1 -- 172.21.15.170:0/1269200896 shutdown_connections 2024-03-29T11:13:26.138 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.138+0000 7f152a6f1640 1 -- 172.21.15.170:0/1269200896 wait complete. 2024-03-29T11:13:26.138 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.139+0000 7f152a6f1640 10 monclient: build_initial_monmap 2024-03-29T11:13:26.138 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.139+0000 7f152a6f1640 10 monclient: monmap: 2024-03-29T11:13:26.138 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:13:26.138 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:26.138 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:13:26.138 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:13:26.138 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:26.138 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:26.138 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:13:26.138 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:13:26.138 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:13:26.138 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:26.142 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.141+0000 7f152a6f1640 1 librados: starting msgr at 2024-03-29T11:13:26.142 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.141+0000 7f152a6f1640 1 librados: starting objecter 2024-03-29T11:13:26.142 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.141+0000 7f152a6f1640 1 Processor -- start 2024-03-29T11:13:26.142 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.141+0000 7f152a6f1640 1 -- start start 2024-03-29T11:13:26.142 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.141+0000 7f152a6f1640 1 librados: setting wanted keys 2024-03-29T11:13:26.142 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.141+0000 7f152a6f1640 1 librados: calling monclient init 2024-03-29T11:13:26.142 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.141+0000 7f152a6f1640 10 monclient: init 2024-03-29T11:13:26.142 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.142+0000 7f152a6f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:26.143 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.142+0000 7f152a6f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:13:26.143 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.142+0000 7f152a6f1640 10 monclient(hunting): picked mon.noname-a con 0x7f15240a9d70 addr v1:172.21.15.170:6789/0 2024-03-29T11:13:26.143 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.143+0000 7f152a6f1640 10 monclient(hunting): picked mon.noname-b con 0x7f152413f650 addr v1:172.21.15.184:6789/0 2024-03-29T11:13:26.143 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.143+0000 7f152a6f1640 10 monclient(hunting): picked mon.noname-c con 0x7f15241457f0 addr v1:172.21.15.170:6790/0 2024-03-29T11:13:26.143 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.143+0000 7f152a6f1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1524071e10 con 0x7f15240a9d70 2024-03-29T11:13:26.143 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.143+0000 7f152a6f1640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1524070c20 con 0x7f152413f650 2024-03-29T11:13:26.143 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.143+0000 7f152a6f1640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1524070150 con 0x7f15241457f0 2024-03-29T11:13:26.143 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.143+0000 7f152a6f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:26.143 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.143+0000 7f152a6f1640 10 monclient(hunting): authenticate will time out at 987.454998s 2024-03-29T11:13:26.143 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.144+0000 7f1528c8e640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f15241457f0 0x7f15241e64d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:47752/0 (socket says 172.21.15.170:47752) 2024-03-29T11:13:26.143 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.144+0000 7f1528c8e640 1 -- 172.21.15.170:0/3047566445 learned_addr learned my addr 172.21.15.170:0/3047566445 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:13:26.143 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.144+0000 7f1520ff9640 10 client.?.objecter ms_handle_connect 0x7f15241457f0 2024-03-29T11:13:26.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.144+0000 7f1520ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:26.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.144+0000 7f1520ff9640 1 -- 172.21.15.170:0/3047566445 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401828905 0 0) 0x7f1524070150 con 0x7f15241457f0 2024-03-29T11:13:26.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.144+0000 7f1520ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:13:26.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.144+0000 7f1520ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:26.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.144+0000 7f1520ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:26.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.144+0000 7f1520ff9640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:13:26.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.144+0000 7f1520ff9640 1 -- 172.21.15.170:0/3047566445 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1510004eb0 con 0x7f15241457f0 2024-03-29T11:13:26.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.145+0000 7f1520ff9640 10 client.?.objecter ms_handle_connect 0x7f152413f650 2024-03-29T11:13:26.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.145+0000 7f1520ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:26.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.145+0000 7f1520ff9640 1 -- 172.21.15.170:0/3047566445 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3916050295 0 0) 0x7f1510004eb0 con 0x7f15241457f0 2024-03-29T11:13:26.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.145+0000 7f1520ff9640 1 -- 172.21.15.170:0/3047566445 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1524070150 con 0x7f15241457f0 2024-03-29T11:13:26.144 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.145+0000 7f1520ff9640 1 -- 172.21.15.170:0/3047566445 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f151c001c90 con 0x7f15241457f0 2024-03-29T11:13:26.145 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.145+0000 7f1520ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:26.145 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.145+0000 7f1520ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:26.145 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.145+0000 7f1520ff9640 10 monclient(hunting): dump: 2024-03-29T11:13:26.146 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 1 -- 172.21.15.170:0/3047566445 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312020405 0 0) 0x7f1524070c20 con 0x7f152413f650 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:26.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:26.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:13:26.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 1 -- 172.21.15.170:0/3047566445 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1510004eb0 con 0x7f152413f650 2024-03-29T11:13:26.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 10 client.?.objecter ms_handle_connect 0x7f15240a9d70 2024-03-29T11:13:26.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:26.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 1 -- 172.21.15.170:0/3047566445 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2691050587 0 0) 0x7f1524070150 con 0x7f15241457f0 2024-03-29T11:13:26.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 1 -- 172.21.15.170:0/3047566445 >> v1:172.21.15.184:6789/0 conn(0x7f152413f650 legacy=0x7f15241dc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:26.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 1 -- 172.21.15.170:0/3047566445 >> v1:172.21.15.170:6789/0 conn(0x7f15240a9d70 legacy=0x7f15241db980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:26.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:13:26.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 1 monclient: found mon.c 2024-03-29T11:13:26.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:26.149 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:26.149 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 1 -- 172.21.15.170:0/3047566445 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15240702e0 con 0x7f15241457f0 2024-03-29T11:13:26.149 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 10 monclient: _finish_auth 0 2024-03-29T11:13:26.149 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:13:26.149 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:26.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f152a6f1640 5 monclient: authenticate success, global_id 4298 2024-03-29T11:13:26.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f152a6f1640 10 monclient: _renew_subs 2024-03-29T11:13:26.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f152a6f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:26.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f152a6f1640 1 -- 172.21.15.170:0/3047566445 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1524142440 con 0x7f15241457f0 2024-03-29T11:13:26.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f152a6f1640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:26.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f152a6f1640 10 monclient: _renew_subs 2024-03-29T11:13:26.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f152a6f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:26.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f152a6f1640 1 -- 172.21.15.170:0/3047566445 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1524070550 con 0x7f15241457f0 2024-03-29T11:13:26.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f152a6f1640 1 librados: init done 2024-03-29T11:13:26.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 1 -- 172.21.15.170:0/3047566445 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f151c003990 con 0x7f15241457f0 2024-03-29T11:13:26.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:26.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 1 -- 172.21.15.170:0/3047566445 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f151c003da0 con 0x7f15241457f0 2024-03-29T11:13:26.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:26.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:26.151 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f1520ff9640 10 monclient: dump: 2024-03-29T11:13:26.151 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:26.151 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:26.151 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:26.151 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:26.151 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:26.151 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:26.151 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:26.151 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:26.151 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:26.151 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:26.151 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.146+0000 7f15227fc640 4 set_mon_vals no callback set 2024-03-29T11:13:26.151 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.147+0000 7f152a6f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:26.151 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.147+0000 7f152a6f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:26.152 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.147+0000 7f152a6f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:26.152 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.149+0000 7f1520ff9640 1 -- 172.21.15.170:0/3047566445 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f151c003f00 con 0x7f15241457f0 2024-03-29T11:13:26.152 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.147+0000 7f152a6f1640 1 -- 172.21.15.170:0/3047566445 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1504005080 con 0x7f15241457f0 2024-03-29T11:13:26.152 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.149+0000 7f1520ff9640 1 -- 172.21.15.170:0/3047566445 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f151c084d80 con 0x7f15241457f0 2024-03-29T11:13:26.152 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.149+0000 7f1520ff9640 10 client.4298.objecter ms_dispatch 0x7f1524000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:26.152 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.149+0000 7f1520ff9640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:26.152 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.149+0000 7f1520ff9640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:26.152 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.149+0000 7f1520ff9640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-29T11:13:26.152 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.150+0000 7f1523fff640 10 monclient: get_auth_request con 0x7f151006ddc0 auth_method 0 2024-03-29T11:13:26.152 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.150+0000 7f1520ff9640 10 client.4298.objecter ms_handle_connect 0x7f151006ddc0 2024-03-29T11:13:26.153 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.158+0000 7f1520ff9640 1 -- 172.21.15.170:0/3047566445 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f151c050660 con 0x7f15241457f0 2024-03-29T11:13:26.153 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.158+0000 7f1520ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:26.153 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.158+0000 7f1520ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:26.435 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.458+0000 7f152a6f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:26.436 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.458+0000 7f152a6f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:26.436 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.458+0000 7f152a6f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:26.436 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.458+0000 7f152a6f1640 1 -- 172.21.15.170:0/3047566445 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1504005330 con 0x7f15241457f0 2024-03-29T11:13:26.436 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.459+0000 7f1520ff9640 1 -- 172.21.15.170:0/3047566445 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f151c052590 con 0x7f15241457f0 2024-03-29T11:13:26.436 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.459+0000 7f1520ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:26.436 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.459+0000 7f1520ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:26.437 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:13:26.437 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:26.440 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.462+0000 7f152a6f1640 10 librados: watch_flush enter 2024-03-29T11:13:26.440 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.462+0000 7f152a6f1640 10 librados: watch_flush exit 2024-03-29T11:13:26.440 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.462+0000 7f152a6f1640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-29T11:13:26.440 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.462+0000 7f152a6f1640 10 client.4298.objecter successfully canceled tick 2024-03-29T11:13:26.440 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.462+0000 7f152a6f1640 1 -- 172.21.15.170:0/3047566445 >> v1:172.21.15.170:6816/3298994444 conn(0x7f151006ddc0 legacy=0x7f1510070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:26.440 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.462+0000 7f152a6f1640 10 monclient: shutdown 2024-03-29T11:13:26.440 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.462+0000 7f152a6f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:26.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.462+0000 7f152a6f1640 1 -- 172.21.15.170:0/3047566445 >> v1:172.21.15.170:6790/0 conn(0x7f15241457f0 legacy=0x7f15241e64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:26.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.463+0000 7f152a6f1640 1 -- 172.21.15.170:0/3047566445 shutdown_connections 2024-03-29T11:13:26.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.463+0000 7f152a6f1640 1 -- 172.21.15.170:0/3047566445 >> 172.21.15.170:0/3047566445 conn(0x7f1524066a00 msgr2=0x7f15240b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:26.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.463+0000 7f152a6f1640 1 -- 172.21.15.170:0/3047566445 shutdown_connections 2024-03-29T11:13:26.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.463+0000 7f152a6f1640 1 -- 172.21.15.170:0/3047566445 wait complete. 2024-03-29T11:13:26.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:26.463+0000 7f152a6f1640 1 librados: shutdown 2024-03-29T11:13:26.454 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:33.260 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:33.406 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.428+0000 7efdcd356640 10 monclient: get_monmap_and_config 2024-03-29T11:13:33.406 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.428+0000 7efdcd356640 10 monclient: build_initial_monmap 2024-03-29T11:13:33.407 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.428+0000 7efdcd356640 10 monclient: monmap: 2024-03-29T11:13:33.407 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:13:33.407 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:33.407 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:13:33.429647+0000 2024-03-29T11:13:33.407 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:13:33.429647+0000 2024-03-29T11:13:33.407 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:33.407 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:33.407 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:13:33.407 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:13:33.407 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:13:33.407 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:33.408 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.430+0000 7efdcd356640 1 Processor -- start 2024-03-29T11:13:33.408 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.430+0000 7efdcd356640 1 -- start start 2024-03-29T11:13:33.408 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.430+0000 7efdcd356640 10 monclient: init 2024-03-29T11:13:33.409 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.431+0000 7efdcd356640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:33.409 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.431+0000 7efdcd356640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:13:33.410 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.431+0000 7efdcd356640 10 monclient(hunting): picked mon.noname-a con 0x7efdc81457f0 addr v1:172.21.15.170:6789/0 2024-03-29T11:13:33.410 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.431+0000 7efdcd356640 10 monclient(hunting): picked mon.noname-c con 0x7efdc813f650 addr v1:172.21.15.184:6789/0 2024-03-29T11:13:33.411 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.431+0000 7efdcd356640 10 monclient(hunting): picked mon.noname-b con 0x7efdc80a9d70 addr v1:172.21.15.170:6790/0 2024-03-29T11:13:33.411 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.431+0000 7efdcd356640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc80aaef0 con 0x7efdc81457f0 2024-03-29T11:13:33.411 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.431+0000 7efdcd356640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc8145bd0 con 0x7efdc813f650 2024-03-29T11:13:33.411 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.431+0000 7efdcd356640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc8145eb0 con 0x7efdc80a9d70 2024-03-29T11:13:33.411 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.431+0000 7efdcd356640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:33.411 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.431+0000 7efdcd356640 10 monclient(hunting): authenticate will time out at 994.743245s 2024-03-29T11:13:33.411 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.431+0000 7efdc67fc640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7efdc813f650 0x7efdc80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:42242/0 (socket says 172.21.15.170:42242) 2024-03-29T11:13:33.411 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.431+0000 7efdc67fc640 1 -- 172.21.15.170:0/101148859 learned_addr learned my addr 172.21.15.170:0/101148859 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:13:33.413 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.435+0000 7efdc5ffb640 1 -- 172.21.15.170:0/101148859 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613526350 0 0) 0x7efdc8145bd0 con 0x7efdc813f650 2024-03-29T11:13:33.413 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.435+0000 7efdc5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:13:33.414 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.435+0000 7efdc5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:33.414 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.435+0000 7efdc5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:33.414 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.435+0000 7efdc5ffb640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:13:33.414 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.435+0000 7efdc5ffb640 1 -- 172.21.15.170:0/101148859 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdb0003520 con 0x7efdc813f650 2024-03-29T11:13:33.414 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.436+0000 7efdc5ffb640 1 -- 172.21.15.170:0/101148859 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4044748158 0 0) 0x7efdb0003520 con 0x7efdc813f650 2024-03-29T11:13:33.415 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.436+0000 7efdc5ffb640 1 -- 172.21.15.170:0/101148859 >> v1:172.21.15.170:6790/0 conn(0x7efdc80a9d70 legacy=0x7efdc814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:33.415 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.436+0000 7efdc5ffb640 1 -- 172.21.15.170:0/101148859 >> v1:172.21.15.170:6789/0 conn(0x7efdc81457f0 legacy=0x7efdc813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:13:33.417 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.437+0000 7efdc5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:13:33.417 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.437+0000 7efdc5ffb640 1 monclient: found mon.noname-c 2024-03-29T11:13:33.417 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.437+0000 7efdc5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:33.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.437+0000 7efdc5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:13:33.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.437+0000 7efdc5ffb640 1 -- 172.21.15.170:0/101148859 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc8146340 con 0x7efdc813f650 2024-03-29T11:13:33.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.437+0000 7efdc5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:13:33.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.437+0000 7efdc5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:13:33.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.437+0000 7efdc5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:33.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.437+0000 7efdc5ffb640 1 -- 172.21.15.170:0/101148859 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7efdb8001690 con 0x7efdc813f650 2024-03-29T11:13:33.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.438+0000 7efdcd356640 5 monclient: authenticate success, global_id 4279 2024-03-29T11:13:33.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.438+0000 7efdcd356640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:33.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.438+0000 7efdc5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:33.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.438+0000 7efdc5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:33.419 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.438+0000 7efdc5ffb640 10 monclient: dump: 2024-03-29T11:13:33.419 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:33.419 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:33.419 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:33.419 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:33.419 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:33.419 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:33.419 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:33.419 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:33.419 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:33.419 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:33.420 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.438+0000 7efdcd356640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:33.420 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.439+0000 7efdc5ffb640 1 -- 172.21.15.170:0/101148859 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdb8001920 con 0x7efdc813f650 2024-03-29T11:13:33.420 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.439+0000 7efdc5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:33.420 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.439+0000 7efdc5ffb640 1 -- 172.21.15.170:0/101148859 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7efdb8001cb0 con 0x7efdc813f650 2024-03-29T11:13:33.420 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.439+0000 7efdc5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:33.420 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.439+0000 7efdc5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:33.420 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.439+0000 7efdc5ffb640 10 monclient: dump: 2024-03-29T11:13:33.420 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:33.420 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:33.420 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:33.420 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:33.420 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:33.420 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:33.420 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:33.420 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:33.421 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:33.421 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:33.421 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.439+0000 7efdcd356640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:33.421 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.439+0000 7efdcd356640 4 set_mon_vals no callback set 2024-03-29T11:13:33.422 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.439+0000 7efdcd356640 10 monclient: shutdown 2024-03-29T11:13:33.422 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.440+0000 7efdcd356640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:33.422 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.440+0000 7efdcd356640 1 -- 172.21.15.170:0/101148859 >> v1:172.21.15.184:6789/0 conn(0x7efdc813f650 legacy=0x7efdc80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:33.422 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.440+0000 7efdcd356640 1 -- 172.21.15.170:0/101148859 shutdown_connections 2024-03-29T11:13:33.423 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.440+0000 7efdcd356640 1 -- 172.21.15.170:0/101148859 >> 172.21.15.170:0/101148859 conn(0x7efdc8066a00 msgr2=0x7efdc8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:33.423 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.440+0000 7efdcd356640 1 -- 172.21.15.170:0/101148859 shutdown_connections 2024-03-29T11:13:33.423 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.440+0000 7efdcd356640 1 -- 172.21.15.170:0/101148859 wait complete. 2024-03-29T11:13:33.423 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.441+0000 7efdcd356640 10 monclient: build_initial_monmap 2024-03-29T11:13:33.423 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.441+0000 7efdcd356640 10 monclient: monmap: 2024-03-29T11:13:33.423 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:13:33.423 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:33.423 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:13:33.423 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:13:33.423 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:33.423 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:33.423 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:13:33.423 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:13:33.423 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:13:33.423 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:33.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.441+0000 7efdcd356640 1 librados: starting msgr at 2024-03-29T11:13:33.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.441+0000 7efdcd356640 1 librados: starting objecter 2024-03-29T11:13:33.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.441+0000 7efdcd356640 1 Processor -- start 2024-03-29T11:13:33.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.442+0000 7efdcd356640 1 -- start start 2024-03-29T11:13:33.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.442+0000 7efdcd356640 1 librados: setting wanted keys 2024-03-29T11:13:33.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.442+0000 7efdcd356640 1 librados: calling monclient init 2024-03-29T11:13:33.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.442+0000 7efdcd356640 10 monclient: init 2024-03-29T11:13:33.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.442+0000 7efdcd356640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:33.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.442+0000 7efdcd356640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:13:33.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.442+0000 7efdcd356640 10 monclient(hunting): picked mon.noname-c con 0x7efdc80a9d70 addr v1:172.21.15.170:6790/0 2024-03-29T11:13:33.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.442+0000 7efdcd356640 10 monclient(hunting): picked mon.noname-a con 0x7efdc813f650 addr v1:172.21.15.170:6789/0 2024-03-29T11:13:33.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.442+0000 7efdcd356640 10 monclient(hunting): picked mon.noname-b con 0x7efdc81457f0 addr v1:172.21.15.184:6789/0 2024-03-29T11:13:33.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.442+0000 7efdcd356640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc8132a20 con 0x7efdc813f650 2024-03-29T11:13:33.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.442+0000 7efdcd356640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc8142890 con 0x7efdc81457f0 2024-03-29T11:13:33.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.442+0000 7efdcd356640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc8142320 con 0x7efdc80a9d70 2024-03-29T11:13:33.425 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.442+0000 7efdcd356640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:33.425 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.442+0000 7efdcd356640 10 monclient(hunting): authenticate will time out at 994.755002s 2024-03-29T11:13:33.425 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.447+0000 7efdc6ffd640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7efdc80a9d70 0x7efdc813fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:46187/0 (socket says 172.21.15.170:46187) 2024-03-29T11:13:33.425 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.447+0000 7efdc6ffd640 1 -- 172.21.15.170:0/3689552374 learned_addr learned my addr 172.21.15.170:0/3689552374 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:13:33.426 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.447+0000 7efdaf7fe640 10 client.?.objecter ms_handle_connect 0x7efdc80a9d70 2024-03-29T11:13:33.426 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.447+0000 7efdaf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:33.426 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.447+0000 7efdaf7fe640 1 -- 172.21.15.170:0/3689552374 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320741435 0 0) 0x7efdc8142320 con 0x7efdc80a9d70 2024-03-29T11:13:33.426 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.447+0000 7efdaf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:33.426 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.447+0000 7efdaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:33.426 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.447+0000 7efdaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:33.426 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.447+0000 7efdaf7fe640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:13:33.426 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.447+0000 7efdaf7fe640 1 -- 172.21.15.170:0/3689552374 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdb4004eb0 con 0x7efdc80a9d70 2024-03-29T11:13:33.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.448+0000 7efdaf7fe640 10 client.?.objecter ms_handle_connect 0x7efdc81457f0 2024-03-29T11:13:33.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.448+0000 7efdaf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:33.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.448+0000 7efdaf7fe640 1 -- 172.21.15.170:0/3689552374 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2202532842 0 0) 0x7efdb4004eb0 con 0x7efdc80a9d70 2024-03-29T11:13:33.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.448+0000 7efdaf7fe640 1 -- 172.21.15.170:0/3689552374 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdc8142320 con 0x7efdc80a9d70 2024-03-29T11:13:33.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.448+0000 7efdaf7fe640 1 -- 172.21.15.170:0/3689552374 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7efdbc0017d0 con 0x7efdc80a9d70 2024-03-29T11:13:33.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.448+0000 7efdaf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:33.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.448+0000 7efdaf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:33.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.448+0000 7efdaf7fe640 10 monclient(hunting): dump: 2024-03-29T11:13:33.429 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:33.429 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:33.429 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:33.429 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:33.429 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:33.429 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:33.429 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:33.429 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:33.429 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:33.429 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:33.429 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.448+0000 7efdaf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:33.429 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.448+0000 7efdaf7fe640 1 -- 172.21.15.170:0/3689552374 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 245261287 0 0) 0x7efdc8142320 con 0x7efdc80a9d70 2024-03-29T11:13:33.429 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.448+0000 7efdaf7fe640 1 -- 172.21.15.170:0/3689552374 >> v1:172.21.15.184:6789/0 conn(0x7efdc81457f0 legacy=0x7efdc81e1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:33.429 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.448+0000 7efdaf7fe640 1 -- 172.21.15.170:0/3689552374 >> v1:172.21.15.170:6789/0 conn(0x7efdc813f650 legacy=0x7efdc81404b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:13:33.430 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdaf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:13:33.430 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdaf7fe640 1 monclient: found mon.c 2024-03-29T11:13:33.430 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:33.430 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdaf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:33.430 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdaf7fe640 1 -- 172.21.15.170:0/3689552374 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc81424b0 con 0x7efdc80a9d70 2024-03-29T11:13:33.430 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdaf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:13:33.430 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdaf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:13:33.430 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:33.430 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdcd356640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:13:33.430 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdcd356640 10 monclient: _renew_subs 2024-03-29T11:13:33.430 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdcd356640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:33.430 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdcd356640 1 -- 172.21.15.170:0/3689552374 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc8144840 con 0x7efdc80a9d70 2024-03-29T11:13:33.430 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdcd356640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:33.430 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdcd356640 10 monclient: _renew_subs 2024-03-29T11:13:33.431 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdcd356640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:33.431 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdcd356640 1 -- 172.21.15.170:0/3689552374 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc8141c80 con 0x7efdc80a9d70 2024-03-29T11:13:33.431 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdcd356640 1 librados: init done 2024-03-29T11:13:33.431 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdaf7fe640 1 -- 172.21.15.170:0/3689552374 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdbc001a60 con 0x7efdc80a9d70 2024-03-29T11:13:33.431 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdaf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:33.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdaf7fe640 1 -- 172.21.15.170:0/3689552374 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7efdbc004a80 con 0x7efdc80a9d70 2024-03-29T11:13:33.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:33.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdaf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:33.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdc57fa640 4 set_mon_vals no callback set 2024-03-29T11:13:33.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.449+0000 7efdaf7fe640 10 monclient: dump: 2024-03-29T11:13:33.432 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:33.432 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:33.432 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:33.432 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:33.432 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:33.433 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:33.433 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:33.433 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:33.433 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:33.433 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:33.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.451+0000 7efdcd356640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:33.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.451+0000 7efdcd356640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:33.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.451+0000 7efdcd356640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:33.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.451+0000 7efdcd356640 1 -- 172.21.15.170:0/3689552374 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda8005080 con 0x7efdc80a9d70 2024-03-29T11:13:33.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.453+0000 7efdaf7fe640 1 -- 172.21.15.170:0/3689552374 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7efdbc01aad0 con 0x7efdc80a9d70 2024-03-29T11:13:33.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.454+0000 7efdaf7fe640 1 -- 172.21.15.170:0/3689552374 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7efdbc084b80 con 0x7efdc80a9d70 2024-03-29T11:13:33.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.454+0000 7efdaf7fe640 10 client.4301.objecter ms_dispatch 0x7efdc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:33.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.454+0000 7efdaf7fe640 3 client.4301.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:33.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.454+0000 7efdaf7fe640 3 client.4301.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:33.434 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.454+0000 7efdaf7fe640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-29T11:13:33.434 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.454+0000 7efdc67fc640 10 monclient: get_auth_request con 0x7efdb406dbd0 auth_method 0 2024-03-29T11:13:33.434 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.455+0000 7efdaf7fe640 10 client.4301.objecter ms_handle_connect 0x7efdb406dbd0 2024-03-29T11:13:33.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.463+0000 7efdaf7fe640 1 -- 172.21.15.170:0/3689552374 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efdbc050290 con 0x7efdc80a9d70 2024-03-29T11:13:33.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.463+0000 7efdaf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:33.442 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.463+0000 7efdaf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:33.762 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.783+0000 7efdcd356640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:33.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.783+0000 7efdcd356640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:33.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.783+0000 7efdcd356640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:33.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.783+0000 7efdcd356640 1 -- 172.21.15.170:0/3689552374 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efda8005330 con 0x7efdc80a9d70 2024-03-29T11:13:33.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.785+0000 7efdaf7fe640 1 -- 172.21.15.170:0/3689552374 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7efdbc052390 con 0x7efdc80a9d70 2024-03-29T11:13:33.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.785+0000 7efdaf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:33.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.785+0000 7efdaf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:33.764 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:13:33.764 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:33.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.788+0000 7efdcd356640 10 librados: watch_flush enter 2024-03-29T11:13:33.767 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.788+0000 7efdcd356640 10 librados: watch_flush exit 2024-03-29T11:13:33.767 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.788+0000 7efdcd356640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-29T11:13:33.767 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.788+0000 7efdcd356640 10 client.4301.objecter successfully canceled tick 2024-03-29T11:13:33.767 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.788+0000 7efdcd356640 1 -- 172.21.15.170:0/3689552374 >> v1:172.21.15.170:6816/3298994444 conn(0x7efdb406dbd0 legacy=0x7efdb4070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:33.767 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.788+0000 7efdcd356640 10 monclient: shutdown 2024-03-29T11:13:33.767 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.788+0000 7efdcd356640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:33.767 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.788+0000 7efdcd356640 1 -- 172.21.15.170:0/3689552374 >> v1:172.21.15.170:6790/0 conn(0x7efdc80a9d70 legacy=0x7efdc813fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:33.768 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.789+0000 7efdcd356640 1 -- 172.21.15.170:0/3689552374 shutdown_connections 2024-03-29T11:13:33.768 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.789+0000 7efdcd356640 1 -- 172.21.15.170:0/3689552374 >> 172.21.15.170:0/3689552374 conn(0x7efdc8066a00 msgr2=0x7efdc80b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:33.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.789+0000 7efdcd356640 1 -- 172.21.15.170:0/3689552374 shutdown_connections 2024-03-29T11:13:33.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.789+0000 7efdcd356640 1 -- 172.21.15.170:0/3689552374 wait complete. 2024-03-29T11:13:33.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:33.789+0000 7efdcd356640 1 librados: shutdown 2024-03-29T11:13:33.786 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:40.594 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:40.748 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.769+0000 7fa083c4a640 10 monclient: get_monmap_and_config 2024-03-29T11:13:40.748 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.769+0000 7fa083c4a640 10 monclient: build_initial_monmap 2024-03-29T11:13:40.748 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.769+0000 7fa083c4a640 10 monclient: monmap: 2024-03-29T11:13:40.748 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:13:40.749 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:40.749 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:13:40.770999+0000 2024-03-29T11:13:40.749 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:13:40.770999+0000 2024-03-29T11:13:40.749 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:40.749 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:40.749 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:13:40.749 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:13:40.749 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:13:40.749 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:40.750 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.771+0000 7fa083c4a640 1 Processor -- start 2024-03-29T11:13:40.750 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.771+0000 7fa083c4a640 1 -- start start 2024-03-29T11:13:40.750 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.771+0000 7fa083c4a640 10 monclient: init 2024-03-29T11:13:40.753 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.772+0000 7fa083c4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:40.753 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.772+0000 7fa083c4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:13:40.753 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.772+0000 7fa083c4a640 10 monclient(hunting): picked mon.noname-c con 0x7fa07c0ab170 addr v1:172.21.15.184:6789/0 2024-03-29T11:13:40.753 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.772+0000 7fa083c4a640 10 monclient(hunting): picked mon.noname-b con 0x7fa07c13f4a0 addr v1:172.21.15.170:6790/0 2024-03-29T11:13:40.754 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.772+0000 7fa083c4a640 10 monclient(hunting): picked mon.noname-a con 0x7fa07c0aeff0 addr v1:172.21.15.170:6789/0 2024-03-29T11:13:40.754 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.772+0000 7fa083c4a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07c13f010 con 0x7fa07c0aeff0 2024-03-29T11:13:40.754 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.772+0000 7fa083c4a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07c0ab6c0 con 0x7fa07c0ab170 2024-03-29T11:13:40.754 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.772+0000 7fa083c4a640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07c0ab9a0 con 0x7fa07c13f4a0 2024-03-29T11:13:40.754 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.772+0000 7fa083c4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:40.754 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.772+0000 7fa083c4a640 10 monclient(hunting): authenticate will time out at 1002.085010s 2024-03-29T11:13:40.754 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.776+0000 7fa0819e6640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fa07c0ab170 0x7fa07c13c3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:50333/0 (socket says 172.21.15.170:50333) 2024-03-29T11:13:40.755 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.776+0000 7fa0819e6640 1 -- 172.21.15.170:0/925291760 learned_addr learned my addr 172.21.15.170:0/925291760 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:13:40.756 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.778+0000 7fa0809e4640 1 -- 172.21.15.170:0/925291760 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748527788 0 0) 0x7fa07c13f010 con 0x7fa07c0aeff0 2024-03-29T11:13:40.756 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.778+0000 7fa0809e4640 10 monclient(hunting): _negotiate 2024-03-29T11:13:40.756 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.778+0000 7fa0809e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:40.756 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.778+0000 7fa0809e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:40.756 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.778+0000 7fa0809e4640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:13:40.756 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.778+0000 7fa0809e4640 1 -- 172.21.15.170:0/925291760 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa068003520 con 0x7fa07c0aeff0 2024-03-29T11:13:40.758 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.780+0000 7fa0809e4640 1 -- 172.21.15.170:0/925291760 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361888492 0 0) 0x7fa07c0ab6c0 con 0x7fa07c0ab170 2024-03-29T11:13:40.764 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.780+0000 7fa0809e4640 10 monclient(hunting): _negotiate 2024-03-29T11:13:40.764 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.780+0000 7fa0809e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:40.764 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.780+0000 7fa0809e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:40.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.780+0000 7fa0809e4640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:13:40.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.780+0000 7fa0809e4640 1 -- 172.21.15.170:0/925291760 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa07c13f010 con 0x7fa07c0ab170 2024-03-29T11:13:40.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa0809e4640 1 -- 172.21.15.170:0/925291760 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 717870667 0 0) 0x7fa068003520 con 0x7fa07c0aeff0 2024-03-29T11:13:40.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa0809e4640 1 -- 172.21.15.170:0/925291760 >> v1:172.21.15.170:6790/0 conn(0x7fa07c13f4a0 legacy=0x7fa07c13f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:40.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa0809e4640 1 -- 172.21.15.170:0/925291760 >> v1:172.21.15.184:6789/0 conn(0x7fa07c0ab170 legacy=0x7fa07c13c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:40.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa0809e4640 10 monclient: _finish_hunting 0 2024-03-29T11:13:40.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa0809e4640 1 monclient: found mon.noname-a 2024-03-29T11:13:40.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa0809e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:40.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa0809e4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:13:40.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa0809e4640 1 -- 172.21.15.170:0/925291760 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07c145830 con 0x7fa07c0aeff0 2024-03-29T11:13:40.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa0809e4640 10 monclient: _finish_auth 0 2024-03-29T11:13:40.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa0809e4640 10 monclient: _check_auth_tickets 2024-03-29T11:13:40.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa0809e4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:40.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa083c4a640 5 monclient: authenticate success, global_id 4293 2024-03-29T11:13:40.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa083c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:40.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa0809e4640 1 -- 172.21.15.170:0/925291760 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa078001690 con 0x7fa07c0aeff0 2024-03-29T11:13:40.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa0809e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:40.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa0809e4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:40.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa0809e4640 10 monclient: dump: 2024-03-29T11:13:40.766 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:40.766 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:40.766 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:40.767 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:40.767 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:40.767 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:40.767 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:40.767 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:40.768 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:40.768 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:40.768 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa083c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:40.768 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa0809e4640 1 -- 172.21.15.170:0/925291760 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0780017f0 con 0x7fa07c0aeff0 2024-03-29T11:13:40.768 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa0809e4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:40.768 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa083c4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:40.768 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.783+0000 7fa083c4a640 4 set_mon_vals no callback set 2024-03-29T11:13:40.768 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.784+0000 7fa083c4a640 10 monclient: shutdown 2024-03-29T11:13:40.768 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.784+0000 7fa083c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:40.768 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.784+0000 7fa083c4a640 1 -- 172.21.15.170:0/925291760 >> v1:172.21.15.170:6789/0 conn(0x7fa07c0aeff0 legacy=0x7fa07c14ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:40.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.787+0000 7fa083c4a640 1 -- 172.21.15.170:0/925291760 shutdown_connections 2024-03-29T11:13:40.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.787+0000 7fa083c4a640 1 -- 172.21.15.170:0/925291760 >> 172.21.15.170:0/925291760 conn(0x7fa07c0b35e0 msgr2=0x7fa07c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:40.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.788+0000 7fa083c4a640 1 -- 172.21.15.170:0/925291760 shutdown_connections 2024-03-29T11:13:40.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.788+0000 7fa083c4a640 1 -- 172.21.15.170:0/925291760 wait complete. 2024-03-29T11:13:40.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.789+0000 7fa083c4a640 10 monclient: build_initial_monmap 2024-03-29T11:13:40.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.789+0000 7fa083c4a640 10 monclient: monmap: 2024-03-29T11:13:40.771 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:13:40.771 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:40.772 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:13:40.772 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:13:40.772 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:40.772 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:40.772 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:13:40.772 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:13:40.772 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:13:40.772 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:40.772 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.789+0000 7fa083c4a640 1 librados: starting msgr at 2024-03-29T11:13:40.772 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.789+0000 7fa083c4a640 1 librados: starting objecter 2024-03-29T11:13:40.772 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.789+0000 7fa083c4a640 1 Processor -- start 2024-03-29T11:13:40.772 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.790+0000 7fa083c4a640 1 -- start start 2024-03-29T11:13:40.772 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.790+0000 7fa083c4a640 1 librados: setting wanted keys 2024-03-29T11:13:40.772 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.790+0000 7fa083c4a640 1 librados: calling monclient init 2024-03-29T11:13:40.772 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.790+0000 7fa083c4a640 10 monclient: init 2024-03-29T11:13:40.773 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.791+0000 7fa083c4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:40.773 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.791+0000 7fa083c4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:13:40.773 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.791+0000 7fa083c4a640 10 monclient(hunting): picked mon.noname-b con 0x7fa07c0ab170 addr v1:172.21.15.184:6789/0 2024-03-29T11:13:40.773 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.791+0000 7fa083c4a640 10 monclient(hunting): picked mon.noname-a con 0x7fa07c0aeff0 addr v1:172.21.15.170:6789/0 2024-03-29T11:13:40.773 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.791+0000 7fa083c4a640 10 monclient(hunting): picked mon.noname-c con 0x7fa07c13f4a0 addr v1:172.21.15.170:6790/0 2024-03-29T11:13:40.773 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.791+0000 7fa083c4a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07c150cc0 con 0x7fa07c0aeff0 2024-03-29T11:13:40.773 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.791+0000 7fa083c4a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07c1413e0 con 0x7fa07c0ab170 2024-03-29T11:13:40.773 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.791+0000 7fa083c4a640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07c1340b0 con 0x7fa07c13f4a0 2024-03-29T11:13:40.773 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.791+0000 7fa083c4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:40.773 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.791+0000 7fa083c4a640 10 monclient(hunting): authenticate will time out at 1002.103246s 2024-03-29T11:13:40.774 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.794+0000 7fa0819e6640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fa07c0ab170 0x7fa07c151200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:50445/0 (socket says 172.21.15.170:50445) 2024-03-29T11:13:40.774 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.794+0000 7fa0819e6640 1 -- 172.21.15.170:0/4092196362 learned_addr learned my addr 172.21.15.170:0/4092196362 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:13:40.775 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.797+0000 7fa0727fc640 10 client.?.objecter ms_handle_connect 0x7fa07c0aeff0 2024-03-29T11:13:40.775 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.797+0000 7fa0727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:40.775 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.797+0000 7fa0727fc640 10 client.?.objecter ms_handle_connect 0x7fa07c0ab170 2024-03-29T11:13:40.775 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.797+0000 7fa0727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:40.777 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 10 client.?.objecter ms_handle_connect 0x7fa07c13f4a0 2024-03-29T11:13:40.777 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:40.777 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584561402 0 0) 0x7fa07c1413e0 con 0x7fa07c0ab170 2024-03-29T11:13:40.777 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:40.777 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:40.777 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:40.778 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:13:40.778 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa060004eb0 con 0x7fa07c0ab170 2024-03-29T11:13:40.778 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265167263 0 0) 0x7fa07c150cc0 con 0x7fa07c0aeff0 2024-03-29T11:13:40.778 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:40.778 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:40.778 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:40.778 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:13:40.778 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa07c1413e0 con 0x7fa07c0aeff0 2024-03-29T11:13:40.778 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176764692 0 0) 0x7fa07c1340b0 con 0x7fa07c13f4a0 2024-03-29T11:13:40.778 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:40.778 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:40.778 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:40.778 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:13:40.778 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa07c150cc0 con 0x7fa07c13f4a0 2024-03-29T11:13:40.779 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 510813165 0 0) 0x7fa07c150cc0 con 0x7fa07c13f4a0 2024-03-29T11:13:40.779 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa07c1340b0 con 0x7fa07c13f4a0 2024-03-29T11:13:40.779 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa078003690 con 0x7fa07c13f4a0 2024-03-29T11:13:40.779 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:40.779 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:40.779 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 10 monclient(hunting): dump: 2024-03-29T11:13:40.779 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:40.779 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:40.779 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:40.779 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:40.779 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:40.779 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:40.779 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:40.779 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:40.780 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:40.780 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:40.780 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.798+0000 7fa0727fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:40.780 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.799+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 223917117 0 0) 0x7fa060004eb0 con 0x7fa07c0ab170 2024-03-29T11:13:40.780 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.799+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa07c150cc0 con 0x7fa07c0ab170 2024-03-29T11:13:40.783 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.800+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2798269038 0 0) 0x7fa07c1413e0 con 0x7fa07c0aeff0 2024-03-29T11:13:40.783 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.800+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa060004eb0 con 0x7fa07c0aeff0 2024-03-29T11:13:40.783 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.800+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2691819500 0 0) 0x7fa07c1340b0 con 0x7fa07c13f4a0 2024-03-29T11:13:40.784 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.800+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 >> v1:172.21.15.184:6789/0 conn(0x7fa07c0ab170 legacy=0x7fa07c151200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:40.784 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.800+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 >> v1:172.21.15.170:6789/0 conn(0x7fa07c0aeff0 legacy=0x7fa07c14f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:40.784 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.800+0000 7fa0727fc640 10 monclient: _finish_hunting 0 2024-03-29T11:13:40.784 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.800+0000 7fa0727fc640 1 monclient: found mon.c 2024-03-29T11:13:40.784 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.800+0000 7fa0727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:40.784 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.800+0000 7fa0727fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:40.784 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.801+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07c1451c0 con 0x7fa07c13f4a0 2024-03-29T11:13:40.784 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.801+0000 7fa0727fc640 10 monclient: _finish_auth 0 2024-03-29T11:13:40.784 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.801+0000 7fa0727fc640 10 monclient: _check_auth_tickets 2024-03-29T11:13:40.784 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.801+0000 7fa0727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:40.784 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.802+0000 7fa083c4a640 5 monclient: authenticate success, global_id 4304 2024-03-29T11:13:40.784 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.802+0000 7fa083c4a640 10 monclient: _renew_subs 2024-03-29T11:13:40.784 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.802+0000 7fa083c4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:40.784 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.802+0000 7fa083c4a640 1 -- 172.21.15.170:0/4092196362 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa07c145470 con 0x7fa07c13f4a0 2024-03-29T11:13:40.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.802+0000 7fa083c4a640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:40.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.802+0000 7fa083c4a640 10 monclient: _renew_subs 2024-03-29T11:13:40.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.802+0000 7fa083c4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:40.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.802+0000 7fa083c4a640 1 -- 172.21.15.170:0/4092196362 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa07c151af0 con 0x7fa07c13f4a0 2024-03-29T11:13:40.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.802+0000 7fa083c4a640 1 librados: init done 2024-03-29T11:13:40.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.803+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa078001a60 con 0x7fa07c13f4a0 2024-03-29T11:13:40.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.803+0000 7fa0727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:40.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.803+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa078003c40 con 0x7fa07c13f4a0 2024-03-29T11:13:40.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.803+0000 7fa0727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:40.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.803+0000 7fa0727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:40.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.803+0000 7fa0727fc640 10 monclient: dump: 2024-03-29T11:13:40.785 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:40.785 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:40.785 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:40.786 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:40.786 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:40.786 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:40.786 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:40.786 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:40.786 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:40.786 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:40.786 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.803+0000 7fa073fff640 4 set_mon_vals no callback set 2024-03-29T11:13:40.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.804+0000 7fa083c4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:40.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.804+0000 7fa083c4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:40.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.804+0000 7fa083c4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:40.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.804+0000 7fa083c4a640 1 -- 172.21.15.170:0/4092196362 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa064005080 con 0x7fa07c13f4a0 2024-03-29T11:13:40.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.805+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fa078019c10 con 0x7fa07c13f4a0 2024-03-29T11:13:40.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.806+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fa078083850 con 0x7fa07c13f4a0 2024-03-29T11:13:40.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.806+0000 7fa0727fc640 10 client.4304.objecter ms_dispatch 0x7fa07c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:40.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.806+0000 7fa0727fc640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:40.788 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.806+0000 7fa0727fc640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:40.788 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.806+0000 7fa0727fc640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-29T11:13:40.788 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.806+0000 7fa0819e6640 10 monclient: get_auth_request con 0x7fa06006e270 auth_method 0 2024-03-29T11:13:40.788 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.807+0000 7fa0727fc640 10 client.4304.objecter ms_handle_connect 0x7fa06006e270 2024-03-29T11:13:40.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.815+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa07804f130 con 0x7fa07c13f4a0 2024-03-29T11:13:40.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.815+0000 7fa0727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:40.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:40.815+0000 7fa0727fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:41.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.143+0000 7fa083c4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:41.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.143+0000 7fa083c4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:41.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.144+0000 7fa083c4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:41.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.144+0000 7fa083c4a640 1 -- 172.21.15.170:0/4092196362 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa064005330 con 0x7fa07c13f4a0 2024-03-29T11:13:41.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.144+0000 7fa0727fc640 1 -- 172.21.15.170:0/4092196362 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fa078051060 con 0x7fa07c13f4a0 2024-03-29T11:13:41.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.144+0000 7fa0727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:41.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.144+0000 7fa0727fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:41.124 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:13:41.124 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:41.128 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.150+0000 7fa083c4a640 10 librados: watch_flush enter 2024-03-29T11:13:41.129 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.150+0000 7fa083c4a640 10 librados: watch_flush exit 2024-03-29T11:13:41.129 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.150+0000 7fa083c4a640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-29T11:13:41.129 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.150+0000 7fa083c4a640 10 client.4304.objecter successfully canceled tick 2024-03-29T11:13:41.129 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.151+0000 7fa083c4a640 1 -- 172.21.15.170:0/4092196362 >> v1:172.21.15.170:6816/3298994444 conn(0x7fa06006e270 legacy=0x7fa060070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:41.129 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.151+0000 7fa083c4a640 10 monclient: shutdown 2024-03-29T11:13:41.129 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.151+0000 7fa083c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:41.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.151+0000 7fa083c4a640 1 -- 172.21.15.170:0/4092196362 >> v1:172.21.15.170:6790/0 conn(0x7fa07c13f4a0 legacy=0x7fa07c1e6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:41.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.151+0000 7fa083c4a640 1 -- 172.21.15.170:0/4092196362 shutdown_connections 2024-03-29T11:13:41.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.151+0000 7fa083c4a640 1 -- 172.21.15.170:0/4092196362 >> 172.21.15.170:0/4092196362 conn(0x7fa07c0b35e0 msgr2=0x7fa07c0b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:41.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.151+0000 7fa083c4a640 1 -- 172.21.15.170:0/4092196362 shutdown_connections 2024-03-29T11:13:41.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.151+0000 7fa083c4a640 1 -- 172.21.15.170:0/4092196362 wait complete. 2024-03-29T11:13:41.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:41.151+0000 7fa083c4a640 1 librados: shutdown 2024-03-29T11:13:41.147 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:47.958 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:48.111 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.133+0000 7ff89cdc3640 10 monclient: get_monmap_and_config 2024-03-29T11:13:48.112 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.133+0000 7ff89cdc3640 10 monclient: build_initial_monmap 2024-03-29T11:13:48.112 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.133+0000 7ff89cdc3640 10 monclient: monmap: 2024-03-29T11:13:48.112 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:13:48.112 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:48.112 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:13:48.135037+0000 2024-03-29T11:13:48.112 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:13:48.135037+0000 2024-03-29T11:13:48.112 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:48.112 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:48.112 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:13:48.112 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:13:48.113 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:13:48.113 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:48.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.137+0000 7ff89cdc3640 1 Processor -- start 2024-03-29T11:13:48.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.138+0000 7ff89cdc3640 1 -- start start 2024-03-29T11:13:48.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.138+0000 7ff89cdc3640 10 monclient: init 2024-03-29T11:13:48.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.138+0000 7ff89cdc3640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:48.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.138+0000 7ff89cdc3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:13:48.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.138+0000 7ff89cdc3640 10 monclient(hunting): picked mon.noname-b con 0x7ff890006a70 addr v1:172.21.15.170:6790/0 2024-03-29T11:13:48.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.138+0000 7ff89cdc3640 10 monclient(hunting): picked mon.noname-a con 0x7ff89002b100 addr v1:172.21.15.170:6789/0 2024-03-29T11:13:48.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.139+0000 7ff89cdc3640 10 monclient(hunting): picked mon.noname-c con 0x7ff890029c30 addr v1:172.21.15.184:6789/0 2024-03-29T11:13:48.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.139+0000 7ff89cdc3640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89002a270 con 0x7ff89002b100 2024-03-29T11:13:48.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.139+0000 7ff89cdc3640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89002c440 con 0x7ff890029c30 2024-03-29T11:13:48.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.139+0000 7ff89cdc3640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8900c9440 con 0x7ff890006a70 2024-03-29T11:13:48.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.139+0000 7ff89cdc3640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:48.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.139+0000 7ff89cdc3640 10 monclient(hunting): authenticate will time out at 1009.451064s 2024-03-29T11:13:48.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.145+0000 7ff8977fe640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7ff890006a70 0x7ff8900c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:52001/0 (socket says 172.21.15.170:52001) 2024-03-29T11:13:48.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.145+0000 7ff8977fe640 1 -- 172.21.15.170:0/1678534837 learned_addr learned my addr 172.21.15.170:0/1678534837 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:13:48.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.145+0000 7ff896ffd640 1 --1- 172.21.15.170:0/1678534837 >> v1:172.21.15.170:6789/0 conn(0x7ff89002b100 0x7ff8900cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:33553/0 (socket says 172.21.15.170:33553) 2024-03-29T11:13:48.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.146+0000 7ff8967fc640 1 -- 172.21.15.170:0/1678534837 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463987257 0 0) 0x7ff89002c440 con 0x7ff890029c30 2024-03-29T11:13:48.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.146+0000 7ff8967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:48.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.146+0000 7ff8967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:48.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.146+0000 7ff8967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:48.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.146+0000 7ff8967fc640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:13:48.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.146+0000 7ff8967fc640 1 -- 172.21.15.170:0/1678534837 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff888003520 con 0x7ff890029c30 2024-03-29T11:13:48.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.146+0000 7ff8967fc640 1 -- 172.21.15.170:0/1678534837 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589402103 0 0) 0x7ff8900c9440 con 0x7ff890006a70 2024-03-29T11:13:48.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.146+0000 7ff8967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:48.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.146+0000 7ff8967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:48.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.146+0000 7ff8967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:48.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.146+0000 7ff8967fc640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:13:48.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.146+0000 7ff8967fc640 1 -- 172.21.15.170:0/1678534837 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff89002c440 con 0x7ff890006a70 2024-03-29T11:13:48.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 1 -- 172.21.15.170:0/1678534837 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3778686712 0 0) 0x7ff89002c440 con 0x7ff890006a70 2024-03-29T11:13:48.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 1 -- 172.21.15.170:0/1678534837 >> v1:172.21.15.184:6789/0 conn(0x7ff890029c30 legacy=0x7ff8900d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:48.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 1 -- 172.21.15.170:0/1678534837 >> v1:172.21.15.170:6789/0 conn(0x7ff89002b100 legacy=0x7ff8900cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:48.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 10 monclient: _finish_hunting 0 2024-03-29T11:13:48.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 1 monclient: found mon.noname-b 2024-03-29T11:13:48.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:48.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:13:48.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 1 -- 172.21.15.170:0/1678534837 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8900c95d0 con 0x7ff890006a70 2024-03-29T11:13:48.127 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 10 monclient: _finish_auth 0 2024-03-29T11:13:48.127 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 10 monclient: _check_auth_tickets 2024-03-29T11:13:48.127 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:48.127 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 1 -- 172.21.15.170:0/1678534837 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7ff884001610 con 0x7ff890006a70 2024-03-29T11:13:48.127 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:48.127 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:48.127 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 10 monclient: dump: 2024-03-29T11:13:48.127 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:48.127 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:48.127 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:48.127 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:48.127 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:48.127 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:48.127 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:48.127 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:48.127 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:48.128 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:48.128 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff89cdc3640 5 monclient: authenticate success, global_id 4307 2024-03-29T11:13:48.128 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff89cdc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:48.128 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff896ffd640 1 --1- 172.21.15.170:0/1678534837 >> v1:172.21.15.170:6789/0 conn(0x7ff89002b100 0x7ff8900cf100 :-1 s=CLOSED pgs=0 cs=0 l=0).handle_server_banner_and_identify state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-29T11:13:48.128 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 1 -- 172.21.15.170:0/1678534837 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff884001870 con 0x7ff890006a70 2024-03-29T11:13:48.128 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:48.128 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 1 -- 172.21.15.170:0/1678534837 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7ff884001c80 con 0x7ff890006a70 2024-03-29T11:13:48.128 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:48.128 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:48.129 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff8967fc640 10 monclient: dump: 2024-03-29T11:13:48.129 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:48.129 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:48.129 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:48.130 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:48.130 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:48.130 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:48.130 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:48.130 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:48.130 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:48.131 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:48.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff89cdc3640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:48.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.147+0000 7ff89cdc3640 4 set_mon_vals no callback set 2024-03-29T11:13:48.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.148+0000 7ff89cdc3640 10 monclient: shutdown 2024-03-29T11:13:48.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.148+0000 7ff89cdc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:48.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.148+0000 7ff89cdc3640 1 -- 172.21.15.170:0/1678534837 >> v1:172.21.15.170:6790/0 conn(0x7ff890006a70 legacy=0x7ff8900c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:48.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.150+0000 7ff89cdc3640 1 -- 172.21.15.170:0/1678534837 shutdown_connections 2024-03-29T11:13:48.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.150+0000 7ff89cdc3640 1 -- 172.21.15.170:0/1678534837 >> 172.21.15.170:0/1678534837 conn(0x7ff8900058f0 msgr2=0x7ff890005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:48.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.152+0000 7ff89cdc3640 1 -- 172.21.15.170:0/1678534837 shutdown_connections 2024-03-29T11:13:48.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.152+0000 7ff89cdc3640 1 -- 172.21.15.170:0/1678534837 wait complete. 2024-03-29T11:13:48.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.153+0000 7ff89cdc3640 10 monclient: build_initial_monmap 2024-03-29T11:13:48.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.153+0000 7ff89cdc3640 10 monclient: monmap: 2024-03-29T11:13:48.133 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:13:48.133 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:48.133 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:13:48.134 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:13:48.134 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:48.134 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:48.134 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:13:48.134 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:13:48.134 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:13:48.134 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:48.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.153+0000 7ff89cdc3640 1 librados: starting msgr at 2024-03-29T11:13:48.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.153+0000 7ff89cdc3640 1 librados: starting objecter 2024-03-29T11:13:48.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.153+0000 7ff89cdc3640 1 Processor -- start 2024-03-29T11:13:48.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.154+0000 7ff89cdc3640 1 -- start start 2024-03-29T11:13:48.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.154+0000 7ff89cdc3640 1 librados: setting wanted keys 2024-03-29T11:13:48.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.154+0000 7ff89cdc3640 1 librados: calling monclient init 2024-03-29T11:13:48.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.154+0000 7ff89cdc3640 10 monclient: init 2024-03-29T11:13:48.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.155+0000 7ff89cdc3640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:48.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.155+0000 7ff89cdc3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:13:48.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.155+0000 7ff89cdc3640 10 monclient(hunting): picked mon.noname-c con 0x7ff890006a70 addr v1:172.21.15.170:6790/0 2024-03-29T11:13:48.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.157+0000 7ff89cdc3640 10 monclient(hunting): picked mon.noname-a con 0x7ff890029c30 addr v1:172.21.15.170:6789/0 2024-03-29T11:13:48.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.157+0000 7ff89cdc3640 10 monclient(hunting): picked mon.noname-b con 0x7ff8900c2e00 addr v1:172.21.15.184:6789/0 2024-03-29T11:13:48.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.157+0000 7ff89cdc3640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89002e280 con 0x7ff890029c30 2024-03-29T11:13:48.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.157+0000 7ff89cdc3640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89002efd0 con 0x7ff8900c2e00 2024-03-29T11:13:48.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.157+0000 7ff89cdc3640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89002f2b0 con 0x7ff890006a70 2024-03-29T11:13:48.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.157+0000 7ff89cdc3640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:48.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.157+0000 7ff89cdc3640 10 monclient(hunting): authenticate will time out at 1009.469076s 2024-03-29T11:13:48.140 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.160+0000 7ff8977fe640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7ff890006a70 0x7ff8900c26f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:52183/0 (socket says 172.21.15.170:52183) 2024-03-29T11:13:48.140 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.160+0000 7ff8977fe640 1 -- 172.21.15.170:0/3230037449 learned_addr learned my addr 172.21.15.170:0/3230037449 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:13:48.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.160+0000 7ff883fff640 10 client.?.objecter ms_handle_connect 0x7ff890006a70 2024-03-29T11:13:48.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.160+0000 7ff883fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:48.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.160+0000 7ff883fff640 1 -- 172.21.15.170:0/3230037449 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4182875753 0 0) 0x7ff89002f2b0 con 0x7ff890006a70 2024-03-29T11:13:48.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.160+0000 7ff883fff640 10 monclient(hunting): _negotiate 2024-03-29T11:13:48.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.160+0000 7ff883fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:48.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.160+0000 7ff883fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:48.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.160+0000 7ff883fff640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:13:48.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.160+0000 7ff883fff640 1 -- 172.21.15.170:0/3230037449 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff87c004eb0 con 0x7ff890006a70 2024-03-29T11:13:48.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.160+0000 7ff883fff640 1 -- 172.21.15.170:0/3230037449 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2470767500 0 0) 0x7ff87c004eb0 con 0x7ff890006a70 2024-03-29T11:13:48.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.160+0000 7ff883fff640 1 -- 172.21.15.170:0/3230037449 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff89002f2b0 con 0x7ff890006a70 2024-03-29T11:13:48.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.160+0000 7ff883fff640 1 -- 172.21.15.170:0/3230037449 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7ff8840015d0 con 0x7ff890006a70 2024-03-29T11:13:48.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.160+0000 7ff883fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:48.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.160+0000 7ff883fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:48.141 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.160+0000 7ff883fff640 10 monclient(hunting): dump: 2024-03-29T11:13:48.142 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:48.142 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:48.142 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:48.142 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:48.142 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:48.142 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:48.142 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:48.142 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:48.142 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:48.142 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:48.142 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.160+0000 7ff883fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:48.142 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.161+0000 7ff883fff640 1 -- 172.21.15.170:0/3230037449 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 980539290 0 0) 0x7ff89002f2b0 con 0x7ff890006a70 2024-03-29T11:13:48.142 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.161+0000 7ff883fff640 1 -- 172.21.15.170:0/3230037449 >> v1:172.21.15.184:6789/0 conn(0x7ff8900c2e00 legacy=0x7ff89016c1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:13:48.146 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.161+0000 7ff883fff640 1 -- 172.21.15.170:0/3230037449 >> v1:172.21.15.170:6789/0 conn(0x7ff890029c30 legacy=0x7ff890160c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:48.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.161+0000 7ff883fff640 10 monclient: _finish_hunting 0 2024-03-29T11:13:48.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.161+0000 7ff883fff640 1 monclient: found mon.c 2024-03-29T11:13:48.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.161+0000 7ff883fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:48.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.161+0000 7ff883fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:48.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.161+0000 7ff883fff640 1 -- 172.21.15.170:0/3230037449 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89002ce50 con 0x7ff890006a70 2024-03-29T11:13:48.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.161+0000 7ff883fff640 10 monclient: _finish_auth 0 2024-03-29T11:13:48.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.161+0000 7ff883fff640 10 monclient: _check_auth_tickets 2024-03-29T11:13:48.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.162+0000 7ff883fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:48.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.162+0000 7ff89cdc3640 5 monclient: authenticate success, global_id 4310 2024-03-29T11:13:48.147 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.162+0000 7ff89cdc3640 10 monclient: _renew_subs 2024-03-29T11:13:48.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.162+0000 7ff89cdc3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:48.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.162+0000 7ff89cdc3640 1 -- 172.21.15.170:0/3230037449 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff89002d0a0 con 0x7ff890006a70 2024-03-29T11:13:48.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.162+0000 7ff89cdc3640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:48.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.162+0000 7ff89cdc3640 10 monclient: _renew_subs 2024-03-29T11:13:48.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.162+0000 7ff89cdc3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:48.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.162+0000 7ff89cdc3640 1 -- 172.21.15.170:0/3230037449 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff890161500 con 0x7ff890006a70 2024-03-29T11:13:48.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.162+0000 7ff89cdc3640 1 librados: init done 2024-03-29T11:13:48.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.164+0000 7ff883fff640 1 -- 172.21.15.170:0/3230037449 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff884004a90 con 0x7ff890006a70 2024-03-29T11:13:48.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.164+0000 7ff883fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:48.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.164+0000 7ff895ffb640 4 set_mon_vals no callback set 2024-03-29T11:13:48.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.165+0000 7ff883fff640 1 -- 172.21.15.170:0/3230037449 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7ff884004ea0 con 0x7ff890006a70 2024-03-29T11:13:48.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.165+0000 7ff883fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:48.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.165+0000 7ff883fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:48.148 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.165+0000 7ff883fff640 10 monclient: dump: 2024-03-29T11:13:48.148 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:48.149 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:48.149 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:48.149 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:48.149 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:48.149 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:48.149 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:48.149 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:48.149 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:48.149 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:48.149 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.165+0000 7ff881ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:48.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.165+0000 7ff881ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:48.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.165+0000 7ff881ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:48.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.165+0000 7ff881ffb640 1 -- 172.21.15.170:0/3230037449 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff878005080 con 0x7ff890006a70 2024-03-29T11:13:48.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.166+0000 7ff883fff640 1 -- 172.21.15.170:0/3230037449 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7ff8840050c0 con 0x7ff890006a70 2024-03-29T11:13:48.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.167+0000 7ff883fff640 1 -- 172.21.15.170:0/3230037449 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7ff884084c60 con 0x7ff890006a70 2024-03-29T11:13:48.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.167+0000 7ff883fff640 10 client.4310.objecter ms_dispatch 0x7ff898000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:48.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.167+0000 7ff883fff640 3 client.4310.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:48.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.167+0000 7ff883fff640 3 client.4310.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:48.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.167+0000 7ff883fff640 20 client.4310.objecter dump_active .. 0 homeless 2024-03-29T11:13:48.150 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.169+0000 7ff896ffd640 10 monclient: get_auth_request con 0x7ff87c06dc70 auth_method 0 2024-03-29T11:13:48.151 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.169+0000 7ff883fff640 10 client.4310.objecter ms_handle_connect 0x7ff87c06dc70 2024-03-29T11:13:48.154 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.176+0000 7ff883fff640 1 -- 172.21.15.170:0/3230037449 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff884050540 con 0x7ff890006a70 2024-03-29T11:13:48.154 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.176+0000 7ff883fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:48.154 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.176+0000 7ff883fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:48.472 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.494+0000 7ff881ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:48.472 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.494+0000 7ff881ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:48.472 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.494+0000 7ff881ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:13:48.472 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.494+0000 7ff881ffb640 1 -- 172.21.15.170:0/3230037449 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff878005330 con 0x7ff890006a70 2024-03-29T11:13:48.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.496+0000 7ff883fff640 1 -- 172.21.15.170:0/3230037449 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7ff884052470 con 0x7ff890006a70 2024-03-29T11:13:48.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.496+0000 7ff883fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:48.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.496+0000 7ff883fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:48.476 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:13:48.476 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:48.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.500+0000 7ff881ffb640 10 librados: watch_flush enter 2024-03-29T11:13:48.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.500+0000 7ff881ffb640 10 librados: watch_flush exit 2024-03-29T11:13:48.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.500+0000 7ff881ffb640 20 client.4310.objecter shutdown clearing up homeless session... 2024-03-29T11:13:48.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.500+0000 7ff881ffb640 10 client.4310.objecter successfully canceled tick 2024-03-29T11:13:48.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.500+0000 7ff881ffb640 1 -- 172.21.15.170:0/3230037449 >> v1:172.21.15.170:6816/3298994444 conn(0x7ff87c06dc70 legacy=0x7ff87c070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:48.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.500+0000 7ff881ffb640 10 monclient: shutdown 2024-03-29T11:13:48.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.500+0000 7ff881ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:48.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.500+0000 7ff881ffb640 1 -- 172.21.15.170:0/3230037449 >> v1:172.21.15.170:6790/0 conn(0x7ff890006a70 legacy=0x7ff8900c26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:48.481 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.500+0000 7ff881ffb640 1 -- 172.21.15.170:0/3230037449 shutdown_connections 2024-03-29T11:13:48.481 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.500+0000 7ff881ffb640 1 -- 172.21.15.170:0/3230037449 >> 172.21.15.170:0/3230037449 conn(0x7ff8900058f0 msgr2=0x7ff8900c3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:48.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.504+0000 7ff881ffb640 1 -- 172.21.15.170:0/3230037449 shutdown_connections 2024-03-29T11:13:48.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.504+0000 7ff881ffb640 1 -- 172.21.15.170:0/3230037449 wait complete. 2024-03-29T11:13:48.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:48.504+0000 7ff881ffb640 1 librados: shutdown 2024-03-29T11:13:48.502 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:55.310 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:55.465 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.485+0000 7f9582a22640 10 monclient: get_monmap_and_config 2024-03-29T11:13:55.465 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.485+0000 7f9582a22640 10 monclient: build_initial_monmap 2024-03-29T11:13:55.465 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.485+0000 7f9582a22640 10 monclient: monmap: 2024-03-29T11:13:55.465 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:13:55.466 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:55.466 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:13:55.486817+0000 2024-03-29T11:13:55.466 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:13:55.486817+0000 2024-03-29T11:13:55.466 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:55.466 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:55.466 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:13:55.466 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:13:55.466 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:13:55.466 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:55.466 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.486+0000 7f9582a22640 1 Processor -- start 2024-03-29T11:13:55.467 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.486+0000 7f9582a22640 1 -- start start 2024-03-29T11:13:55.467 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.486+0000 7f9582a22640 10 monclient: init 2024-03-29T11:13:55.467 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.488+0000 7f9582a22640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:55.468 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.488+0000 7f9582a22640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:13:55.468 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.488+0000 7f9582a22640 10 monclient(hunting): picked mon.noname-a con 0x7f957c0af5a0 addr v1:172.21.15.170:6789/0 2024-03-29T11:13:55.468 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.488+0000 7f9582a22640 10 monclient(hunting): picked mon.noname-c con 0x7f957c13bf10 addr v1:172.21.15.184:6789/0 2024-03-29T11:13:55.468 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.488+0000 7f9582a22640 10 monclient(hunting): picked mon.noname-b con 0x7f957c0ab480 addr v1:172.21.15.170:6790/0 2024-03-29T11:13:55.468 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.488+0000 7f9582a22640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f957c0a94e0 con 0x7f957c0af5a0 2024-03-29T11:13:55.468 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.488+0000 7f9582a22640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f957c0ace10 con 0x7f957c13bf10 2024-03-29T11:13:55.468 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.488+0000 7f9582a22640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f957c0ad0f0 con 0x7f957c0ab480 2024-03-29T11:13:55.468 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.488+0000 7f9582a22640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:55.468 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.488+0000 7f9582a22640 10 monclient(hunting): authenticate will time out at 1016.800933s 2024-03-29T11:13:55.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.491+0000 7f9581a20640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f957c0af5a0 0x7f957c0ad510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:40893/0 (socket says 172.21.15.170:40893) 2024-03-29T11:13:55.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.491+0000 7f9581a20640 1 -- 172.21.15.170:0/3482338762 learned_addr learned my addr 172.21.15.170:0/3482338762 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:13:55.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.495+0000 7f9580a1e640 1 -- 172.21.15.170:0/3482338762 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817291004 0 0) 0x7f957c0ace10 con 0x7f957c13bf10 2024-03-29T11:13:55.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.495+0000 7f9580a1e640 10 monclient(hunting): _negotiate 2024-03-29T11:13:55.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.495+0000 7f9580a1e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:55.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.495+0000 7f9580a1e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:55.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.495+0000 7f9580a1e640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:13:55.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.495+0000 7f9580a1e640 1 -- 172.21.15.170:0/3482338762 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9568005730 con 0x7f957c13bf10 2024-03-29T11:13:55.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.495+0000 7f9580a1e640 1 -- 172.21.15.170:0/3482338762 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570944032 0 0) 0x7f957c0ad0f0 con 0x7f957c0ab480 2024-03-29T11:13:55.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.495+0000 7f9580a1e640 10 monclient(hunting): _negotiate 2024-03-29T11:13:55.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.495+0000 7f9580a1e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:55.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.495+0000 7f9580a1e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:55.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.495+0000 7f9580a1e640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:13:55.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.495+0000 7f9580a1e640 1 -- 172.21.15.170:0/3482338762 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f957c0ace10 con 0x7f957c0ab480 2024-03-29T11:13:55.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.495+0000 7f9580a1e640 1 -- 172.21.15.170:0/3482338762 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860253982 0 0) 0x7f957c0a94e0 con 0x7f957c0af5a0 2024-03-29T11:13:55.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.495+0000 7f9580a1e640 10 monclient(hunting): _negotiate 2024-03-29T11:13:55.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.495+0000 7f9580a1e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:55.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.495+0000 7f9580a1e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:55.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.495+0000 7f9580a1e640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:13:55.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.496+0000 7f9580a1e640 1 -- 172.21.15.170:0/3482338762 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f957c0ad0f0 con 0x7f957c0af5a0 2024-03-29T11:13:55.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.496+0000 7f9580a1e640 1 -- 172.21.15.170:0/3482338762 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3117834395 0 0) 0x7f957c0ace10 con 0x7f957c0ab480 2024-03-29T11:13:55.481 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.496+0000 7f9580a1e640 1 -- 172.21.15.170:0/3482338762 >> v1:172.21.15.184:6789/0 conn(0x7f957c13bf10 legacy=0x7f957c13c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:55.481 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.496+0000 7f9580a1e640 1 -- 172.21.15.170:0/3482338762 >> v1:172.21.15.170:6789/0 conn(0x7f957c0af5a0 legacy=0x7f957c0ad510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:55.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.496+0000 7f9580a1e640 10 monclient: _finish_hunting 0 2024-03-29T11:13:55.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.496+0000 7f9580a1e640 1 monclient: found mon.noname-b 2024-03-29T11:13:55.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.496+0000 7f9580a1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:55.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.496+0000 7f9580a1e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:13:55.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.496+0000 7f9580a1e640 1 -- 172.21.15.170:0/3482338762 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957c0ade00 con 0x7f957c0ab480 2024-03-29T11:13:55.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.496+0000 7f9580a1e640 10 monclient: _finish_auth 0 2024-03-29T11:13:55.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.496+0000 7f9580a1e640 10 monclient: _check_auth_tickets 2024-03-29T11:13:55.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.496+0000 7f9580a1e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:55.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.496+0000 7f9580a1e640 1 -- 172.21.15.170:0/3482338762 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f95740019f0 con 0x7f957c0ab480 2024-03-29T11:13:55.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.496+0000 7f9582a22640 5 monclient: authenticate success, global_id 4313 2024-03-29T11:13:55.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.496+0000 7f9582a22640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:55.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.496+0000 7f9580a1e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:55.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.496+0000 7f9580a1e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:55.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.496+0000 7f9580a1e640 10 monclient: dump: 2024-03-29T11:13:55.483 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:55.483 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:55.483 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:55.483 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:55.483 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:55.483 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:55.483 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:55.483 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:55.483 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:55.483 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:55.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.496+0000 7f9582a22640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:55.486 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.500+0000 7f9580a1e640 1 -- 172.21.15.170:0/3482338762 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9574001c40 con 0x7f957c0ab480 2024-03-29T11:13:55.486 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.500+0000 7f9580a1e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:55.486 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.500+0000 7f9582a22640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:55.486 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.500+0000 7f9582a22640 4 set_mon_vals no callback set 2024-03-29T11:13:55.486 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.501+0000 7f9582a22640 10 monclient: shutdown 2024-03-29T11:13:55.486 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.501+0000 7f9582a22640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:55.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.501+0000 7f9582a22640 1 -- 172.21.15.170:0/3482338762 >> v1:172.21.15.170:6790/0 conn(0x7f957c0ab480 legacy=0x7f957c0ab910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:55.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.501+0000 7f9582a22640 1 -- 172.21.15.170:0/3482338762 shutdown_connections 2024-03-29T11:13:55.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.501+0000 7f9582a22640 1 -- 172.21.15.170:0/3482338762 >> 172.21.15.170:0/3482338762 conn(0x7f957c0673e0 msgr2=0x7f957c0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:55.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.501+0000 7f9582a22640 1 -- 172.21.15.170:0/3482338762 shutdown_connections 2024-03-29T11:13:55.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.501+0000 7f9582a22640 1 -- 172.21.15.170:0/3482338762 wait complete. 2024-03-29T11:13:55.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.502+0000 7f9582a22640 10 monclient: build_initial_monmap 2024-03-29T11:13:55.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.502+0000 7f9582a22640 10 monclient: monmap: 2024-03-29T11:13:55.487 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:13:55.487 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:55.487 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:13:55.487 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:13:55.487 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:55.487 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:55.487 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:13:55.487 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:13:55.488 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:13:55.488 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:55.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.503+0000 7f9582a22640 1 librados: starting msgr at 2024-03-29T11:13:55.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.503+0000 7f9582a22640 1 librados: starting objecter 2024-03-29T11:13:55.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.503+0000 7f9582a22640 1 Processor -- start 2024-03-29T11:13:55.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.503+0000 7f9582a22640 1 -- start start 2024-03-29T11:13:55.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.503+0000 7f9582a22640 1 librados: setting wanted keys 2024-03-29T11:13:55.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.503+0000 7f9582a22640 1 librados: calling monclient init 2024-03-29T11:13:55.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.503+0000 7f9582a22640 10 monclient: init 2024-03-29T11:13:55.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.504+0000 7f9582a22640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:55.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.504+0000 7f9582a22640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:13:55.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.504+0000 7f9582a22640 10 monclient(hunting): picked mon.noname-a con 0x7f957c0ab480 addr v1:172.21.15.170:6789/0 2024-03-29T11:13:55.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.504+0000 7f9582a22640 10 monclient(hunting): picked mon.noname-b con 0x7f957c0af5a0 addr v1:172.21.15.184:6789/0 2024-03-29T11:13:55.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.504+0000 7f9582a22640 10 monclient(hunting): picked mon.noname-c con 0x7f957c13bf10 addr v1:172.21.15.170:6790/0 2024-03-29T11:13:55.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.504+0000 7f9582a22640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f957c13f930 con 0x7f957c0ab480 2024-03-29T11:13:55.489 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.504+0000 7f9582a22640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f957c1d3310 con 0x7f957c0af5a0 2024-03-29T11:13:55.489 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.504+0000 7f9582a22640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f957c0718f0 con 0x7f957c13bf10 2024-03-29T11:13:55.489 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.504+0000 7f9582a22640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:55.489 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.504+0000 7f9582a22640 10 monclient(hunting): authenticate will time out at 1016.816791s 2024-03-29T11:13:55.489 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.509+0000 7f9582221640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f957c13bf10 0x7f957c13d800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:48365/0 (socket says 172.21.15.170:48365) 2024-03-29T11:13:55.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.509+0000 7f9582221640 1 -- 172.21.15.170:0/1788820598 learned_addr learned my addr 172.21.15.170:0/1788820598 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:13:55.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.512+0000 7f95727fc640 10 client.?.objecter ms_handle_connect 0x7f957c13bf10 2024-03-29T11:13:55.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.512+0000 7f95727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:55.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.512+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534633815 0 0) 0x7f957c0718f0 con 0x7f957c13bf10 2024-03-29T11:13:55.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.512+0000 7f95727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:55.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.512+0000 7f95727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:55.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.512+0000 7f95727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:55.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.512+0000 7f95727fc640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:13:55.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.512+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9560004eb0 con 0x7f957c13bf10 2024-03-29T11:13:55.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.512+0000 7f95727fc640 10 client.?.objecter ms_handle_connect 0x7f957c0ab480 2024-03-29T11:13:55.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.512+0000 7f95727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:55.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.512+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181182351 0 0) 0x7f957c13f930 con 0x7f957c0ab480 2024-03-29T11:13:55.494 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.512+0000 7f95727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:55.494 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.512+0000 7f95727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:55.494 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.512+0000 7f95727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:55.494 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.512+0000 7f95727fc640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:13:55.494 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.512+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f957c0718f0 con 0x7f957c0ab480 2024-03-29T11:13:55.494 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.513+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 739889740 0 0) 0x7f957c0718f0 con 0x7f957c0ab480 2024-03-29T11:13:55.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.513+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f957c13f930 con 0x7f957c0ab480 2024-03-29T11:13:55.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.513+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9578001820 con 0x7f957c0ab480 2024-03-29T11:13:55.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.513+0000 7f95727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:55.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.513+0000 7f95727fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:55.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.513+0000 7f95727fc640 10 monclient(hunting): dump: 2024-03-29T11:13:55.495 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:55.495 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:55.495 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:55.495 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:55.495 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:55.495 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:55.495 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:55.495 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:55.496 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:55.496 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:55.496 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.513+0000 7f95727fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:55.496 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.513+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2943377952 0 0) 0x7f9560004eb0 con 0x7f957c13bf10 2024-03-29T11:13:55.496 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.513+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f957c0718f0 con 0x7f957c13bf10 2024-03-29T11:13:55.496 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.513+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9574003bd0 con 0x7f957c13bf10 2024-03-29T11:13:55.496 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.513+0000 7f95727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:55.496 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.513+0000 7f95727fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:55.496 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.513+0000 7f95727fc640 10 monclient(hunting): dump: 2024-03-29T11:13:55.497 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:55.497 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:55.497 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:55.497 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:55.497 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:55.497 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:55.497 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:55.497 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:55.497 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:55.497 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:55.499 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.517+0000 7f95727fc640 10 client.?.objecter ms_handle_connect 0x7f957c0af5a0 2024-03-29T11:13:55.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.517+0000 7f95727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:55.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.517+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2490674586 0 0) 0x7f957c13f930 con 0x7f957c0ab480 2024-03-29T11:13:55.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.517+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 >> v1:172.21.15.170:6790/0 conn(0x7f957c13bf10 legacy=0x7f957c13d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:55.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.517+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 >> v1:172.21.15.184:6789/0 conn(0x7f957c0af5a0 legacy=0x7f957c0700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:55.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.517+0000 7f95727fc640 10 monclient: _finish_hunting 0 2024-03-29T11:13:55.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.517+0000 7f95727fc640 1 monclient: found mon.a 2024-03-29T11:13:55.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.517+0000 7f95727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:55.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.517+0000 7f95727fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:13:55.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.517+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957c070e50 con 0x7f957c0ab480 2024-03-29T11:13:55.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.517+0000 7f95727fc640 10 monclient: _finish_auth 0 2024-03-29T11:13:55.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.517+0000 7f95727fc640 10 monclient: _check_auth_tickets 2024-03-29T11:13:55.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.517+0000 7f95727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:55.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.518+0000 7f9582a22640 5 monclient: authenticate success, global_id 4302 2024-03-29T11:13:55.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.518+0000 7f9582a22640 10 monclient: _renew_subs 2024-03-29T11:13:55.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.518+0000 7f9582a22640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:13:55.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.518+0000 7f9582a22640 1 -- 172.21.15.170:0/1788820598 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f957c071100 con 0x7f957c0ab480 2024-03-29T11:13:55.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.518+0000 7f9582a22640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:55.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.518+0000 7f9582a22640 10 monclient: _renew_subs 2024-03-29T11:13:55.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.518+0000 7f9582a22640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:13:55.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.518+0000 7f9582a22640 1 -- 172.21.15.170:0/1788820598 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f957c0709c0 con 0x7f957c0ab480 2024-03-29T11:13:55.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.518+0000 7f9582a22640 1 librados: init done 2024-03-29T11:13:55.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.520+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9578001d30 con 0x7f957c0ab480 2024-03-29T11:13:55.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.520+0000 7f95727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:55.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.520+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9578004980 con 0x7f957c0ab480 2024-03-29T11:13:55.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.520+0000 7f95727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:55.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.520+0000 7f95727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:55.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.520+0000 7f95727fc640 10 monclient: dump: 2024-03-29T11:13:55.502 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:13:55.502 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:13:55.502 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:55.502 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:13:55.503 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:55.503 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:13:55.503 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:13:55.503 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:13:55.503 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:13:55.503 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:13:55.504 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.522+0000 7f9573fff640 4 set_mon_vals no callback set 2024-03-29T11:13:55.504 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.522+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f957801a950 con 0x7f957c0ab480 2024-03-29T11:13:55.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.523+0000 7f958121f640 10 monclient: get_auth_request con 0x7f956006de40 auth_method 0 2024-03-29T11:13:55.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.523+0000 7f9582a22640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:55.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.523+0000 7f9582a22640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:55.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.523+0000 7f9582a22640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:13:55.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.523+0000 7f9582a22640 1 -- 172.21.15.170:0/1788820598 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9564005080 con 0x7f957c0ab480 2024-03-29T11:13:55.506 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.523+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f9578054790 con 0x7f957c0ab480 2024-03-29T11:13:55.506 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.523+0000 7f95727fc640 10 client.4302.objecter ms_dispatch 0x7f957c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:55.506 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.523+0000 7f95727fc640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:55.506 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.523+0000 7f95727fc640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:55.506 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.524+0000 7f95727fc640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-29T11:13:55.506 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.524+0000 7f95727fc640 10 client.4302.objecter ms_handle_connect 0x7f956006de40 2024-03-29T11:13:55.511 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.532+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9578050a30 con 0x7f957c0ab480 2024-03-29T11:13:55.511 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.533+0000 7f95727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:55.511 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.533+0000 7f95727fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:55.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.852+0000 7f9582a22640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:55.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.852+0000 7f9582a22640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:55.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.852+0000 7f9582a22640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:13:55.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.852+0000 7f9582a22640 1 -- 172.21.15.170:0/1788820598 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9564005330 con 0x7f957c0ab480 2024-03-29T11:13:55.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.856+0000 7f95727fc640 1 -- 172.21.15.170:0/1788820598 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f9578050a30 con 0x7f957c0ab480 2024-03-29T11:13:55.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.856+0000 7f95727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:55.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.856+0000 7f95727fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:55.836 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:13:55.837 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:55.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.860+0000 7f9582a22640 10 librados: watch_flush enter 2024-03-29T11:13:55.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.864+0000 7f9582a22640 10 librados: watch_flush exit 2024-03-29T11:13:55.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.864+0000 7f9582a22640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-29T11:13:55.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.864+0000 7f9582a22640 10 client.4302.objecter successfully canceled tick 2024-03-29T11:13:55.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.864+0000 7f9582a22640 1 -- 172.21.15.170:0/1788820598 >> v1:172.21.15.170:6816/3298994444 conn(0x7f956006de40 legacy=0x7f9560070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:55.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.864+0000 7f9582a22640 10 monclient: shutdown 2024-03-29T11:13:55.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.864+0000 7f9582a22640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:55.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.864+0000 7f9582a22640 1 -- 172.21.15.170:0/1788820598 >> v1:172.21.15.170:6789/0 conn(0x7f957c0ab480 legacy=0x7f957c1d2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:55.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.864+0000 7f9582a22640 1 -- 172.21.15.170:0/1788820598 shutdown_connections 2024-03-29T11:13:55.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.864+0000 7f9582a22640 1 -- 172.21.15.170:0/1788820598 >> 172.21.15.170:0/1788820598 conn(0x7f957c0673e0 msgr2=0x7f957c0ab8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:55.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.864+0000 7f9582a22640 1 -- 172.21.15.170:0/1788820598 shutdown_connections 2024-03-29T11:13:55.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.864+0000 7f9582a22640 1 -- 172.21.15.170:0/1788820598 wait complete. 2024-03-29T11:13:55.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:13:55.864+0000 7f9582a22640 1 librados: shutdown 2024-03-29T11:13:55.864 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:02.670 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:02.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.849+0000 7f87f1d32640 10 monclient: get_monmap_and_config 2024-03-29T11:14:02.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.849+0000 7f87f1d32640 10 monclient: build_initial_monmap 2024-03-29T11:14:02.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.849+0000 7f87f1d32640 10 monclient: monmap: 2024-03-29T11:14:02.827 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:14:02.827 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:02.827 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:14:02.849720+0000 2024-03-29T11:14:02.827 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:14:02.849720+0000 2024-03-29T11:14:02.827 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:02.827 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:02.827 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:14:02.827 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:14:02.827 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:14:02.827 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:02.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.850+0000 7f87f1d32640 1 Processor -- start 2024-03-29T11:14:02.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.850+0000 7f87f1d32640 1 -- start start 2024-03-29T11:14:02.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.850+0000 7f87f1d32640 10 monclient: init 2024-03-29T11:14:02.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.851+0000 7f87f1d32640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:02.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.851+0000 7f87f1d32640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:02.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.851+0000 7f87f1d32640 10 monclient(hunting): picked mon.noname-b con 0x7f87ec0a9ac0 addr v1:172.21.15.170:6790/0 2024-03-29T11:14:02.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.851+0000 7f87f1d32640 10 monclient(hunting): picked mon.noname-a con 0x7f87ec13ee90 addr v1:172.21.15.170:6789/0 2024-03-29T11:14:02.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.851+0000 7f87f1d32640 10 monclient(hunting): picked mon.noname-c con 0x7f87ec0b4100 addr v1:172.21.15.184:6789/0 2024-03-29T11:14:02.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.851+0000 7f87f1d32640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87ec0af6f0 con 0x7f87ec13ee90 2024-03-29T11:14:02.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.851+0000 7f87f1d32640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87ec0aac80 con 0x7f87ec0b4100 2024-03-29T11:14:02.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.851+0000 7f87f1d32640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87ec0aaf60 con 0x7f87ec0a9ac0 2024-03-29T11:14:02.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.851+0000 7f87f1d32640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:02.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.851+0000 7f87f1d32640 10 monclient(hunting): authenticate will time out at 1024.162643s 2024-03-29T11:14:02.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.856+0000 7f87eb7fe640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f87ec0a9ac0 0x7f87ec0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:44439/0 (socket says 172.21.15.170:44439) 2024-03-29T11:14:02.834 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.856+0000 7f87eb7fe640 1 -- 172.21.15.170:0/2855670630 learned_addr learned my addr 172.21.15.170:0/2855670630 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:14:02.835 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.857+0000 7f87ea7fc640 1 -- 172.21.15.170:0/2855670630 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78587613 0 0) 0x7f87ec0aaf60 con 0x7f87ec0a9ac0 2024-03-29T11:14:02.835 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.857+0000 7f87ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:14:02.835 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.857+0000 7f87ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:02.835 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.857+0000 7f87ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:02.835 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.857+0000 7f87ea7fc640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:14:02.835 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.857+0000 7f87ea7fc640 1 -- 172.21.15.170:0/2855670630 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d4003520 con 0x7f87ec0a9ac0 2024-03-29T11:14:02.835 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87ea7fc640 1 -- 172.21.15.170:0/2855670630 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2633721545 0 0) 0x7f87d4003520 con 0x7f87ec0a9ac0 2024-03-29T11:14:02.835 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87ea7fc640 1 -- 172.21.15.170:0/2855670630 >> v1:172.21.15.184:6789/0 conn(0x7f87ec0b4100 legacy=0x7f87ec14a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:02.835 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87ea7fc640 1 -- 172.21.15.170:0/2855670630 >> v1:172.21.15.170:6789/0 conn(0x7f87ec13ee90 legacy=0x7f87ec13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:02.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87ea7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:14:02.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87ea7fc640 1 monclient: found mon.noname-b 2024-03-29T11:14:02.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:02.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87ea7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:14:02.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87ea7fc640 1 -- 172.21.15.170:0/2855670630 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ec13e970 con 0x7f87ec0a9ac0 2024-03-29T11:14:02.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87ea7fc640 10 monclient: _finish_auth 0 2024-03-29T11:14:02.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87ea7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:14:02.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:02.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87f1d32640 5 monclient: authenticate success, global_id 4319 2024-03-29T11:14:02.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87f1d32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:02.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87ea7fc640 1 -- 172.21.15.170:0/2855670630 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f87e4001690 con 0x7f87ec0a9ac0 2024-03-29T11:14:02.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:02.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87ea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:02.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87ea7fc640 10 monclient: dump: 2024-03-29T11:14:02.837 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:02.837 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:02.837 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:02.837 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:02.837 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:02.837 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:02.837 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:02.837 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:02.837 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:02.838 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:02.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87f1d32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:02.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87ea7fc640 1 -- 172.21.15.170:0/2855670630 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87e4001920 con 0x7f87ec0a9ac0 2024-03-29T11:14:02.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:02.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87f1d32640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:02.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.858+0000 7f87f1d32640 4 set_mon_vals no callback set 2024-03-29T11:14:02.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.859+0000 7f87f1d32640 10 monclient: shutdown 2024-03-29T11:14:02.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.859+0000 7f87f1d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:02.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.859+0000 7f87f1d32640 1 -- 172.21.15.170:0/2855670630 >> v1:172.21.15.170:6790/0 conn(0x7f87ec0a9ac0 legacy=0x7f87ec0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:02.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.861+0000 7f87f1d32640 1 -- 172.21.15.170:0/2855670630 shutdown_connections 2024-03-29T11:14:02.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.861+0000 7f87f1d32640 1 -- 172.21.15.170:0/2855670630 >> 172.21.15.170:0/2855670630 conn(0x7f87ec0b35e0 msgr2=0x7f87ec0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:02.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.861+0000 7f87f1d32640 1 -- 172.21.15.170:0/2855670630 shutdown_connections 2024-03-29T11:14:02.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.861+0000 7f87f1d32640 1 -- 172.21.15.170:0/2855670630 wait complete. 2024-03-29T11:14:02.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.862+0000 7f87f1d32640 10 monclient: build_initial_monmap 2024-03-29T11:14:02.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.862+0000 7f87f1d32640 10 monclient: monmap: 2024-03-29T11:14:02.839 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:14:02.839 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:02.840 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:14:02.840 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:14:02.840 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:02.840 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:02.840 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:14:02.840 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:14:02.840 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:14:02.840 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:02.840 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.863+0000 7f87f1d32640 1 librados: starting msgr at 2024-03-29T11:14:02.840 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.863+0000 7f87f1d32640 1 librados: starting objecter 2024-03-29T11:14:02.840 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.863+0000 7f87f1d32640 1 Processor -- start 2024-03-29T11:14:02.841 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.863+0000 7f87f1d32640 1 -- start start 2024-03-29T11:14:02.841 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.863+0000 7f87f1d32640 1 librados: setting wanted keys 2024-03-29T11:14:02.841 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.863+0000 7f87f1d32640 1 librados: calling monclient init 2024-03-29T11:14:02.841 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.863+0000 7f87f1d32640 10 monclient: init 2024-03-29T11:14:02.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.864+0000 7f87f1d32640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:02.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.864+0000 7f87f1d32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:02.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.864+0000 7f87f1d32640 10 monclient(hunting): picked mon.noname-a con 0x7f87ec0a9ac0 addr v1:172.21.15.170:6789/0 2024-03-29T11:14:02.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.864+0000 7f87f1d32640 10 monclient(hunting): picked mon.noname-b con 0x7f87ec0b4100 addr v1:172.21.15.184:6789/0 2024-03-29T11:14:02.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.864+0000 7f87f1d32640 10 monclient(hunting): picked mon.noname-c con 0x7f87ec13ee90 addr v1:172.21.15.170:6790/0 2024-03-29T11:14:02.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.864+0000 7f87f1d32640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87ec06fb10 con 0x7f87ec0a9ac0 2024-03-29T11:14:02.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.864+0000 7f87f1d32640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87ec06f680 con 0x7f87ec0b4100 2024-03-29T11:14:02.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.864+0000 7f87f1d32640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87ec0725e0 con 0x7f87ec13ee90 2024-03-29T11:14:02.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.864+0000 7f87f1d32640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:02.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.864+0000 7f87f1d32640 10 monclient(hunting): authenticate will time out at 1024.175493s 2024-03-29T11:14:02.845 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.868+0000 7f87eaffd640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f87ec0b4100 0x7f87ec1d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:59579/0 (socket says 172.21.15.170:59579) 2024-03-29T11:14:02.845 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.868+0000 7f87eaffd640 1 -- 172.21.15.170:0/1949894321 learned_addr learned my addr 172.21.15.170:0/1949894321 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:14:02.846 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.869+0000 7f87d3fff640 10 client.?.objecter ms_handle_connect 0x7f87ec0b4100 2024-03-29T11:14:02.847 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.869+0000 7f87d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:02.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.870+0000 7f87d3fff640 1 -- 172.21.15.170:0/1949894321 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498772504 0 0) 0x7f87ec06f680 con 0x7f87ec0b4100 2024-03-29T11:14:02.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.870+0000 7f87d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:02.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.870+0000 7f87d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:02.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.870+0000 7f87d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:02.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.870+0000 7f87d3fff640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:14:02.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.870+0000 7f87d3fff640 1 -- 172.21.15.170:0/1949894321 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d8004eb0 con 0x7f87ec0b4100 2024-03-29T11:14:02.851 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.872+0000 7f87d3fff640 10 client.?.objecter ms_handle_connect 0x7f87ec13ee90 2024-03-29T11:14:02.851 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.872+0000 7f87d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:02.851 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.872+0000 7f87d3fff640 1 -- 172.21.15.170:0/1949894321 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211261917 0 0) 0x7f87ec0725e0 con 0x7f87ec13ee90 2024-03-29T11:14:02.851 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.872+0000 7f87d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:02.851 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.872+0000 7f87d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:02.851 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.872+0000 7f87d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:02.851 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.872+0000 7f87d3fff640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:14:02.851 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.872+0000 7f87d3fff640 1 -- 172.21.15.170:0/1949894321 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d80051d0 con 0x7f87ec13ee90 2024-03-29T11:14:02.851 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 1 -- 172.21.15.170:0/1949894321 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3374622825 0 0) 0x7f87d8004eb0 con 0x7f87ec0b4100 2024-03-29T11:14:02.851 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 1 -- 172.21.15.170:0/1949894321 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87ec0725e0 con 0x7f87ec0b4100 2024-03-29T11:14:02.851 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 1 -- 172.21.15.170:0/1949894321 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f87e0001710 con 0x7f87ec0b4100 2024-03-29T11:14:02.852 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:02.852 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:02.852 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 10 monclient(hunting): dump: 2024-03-29T11:14:02.852 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:02.852 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:02.852 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:02.852 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:02.852 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:02.852 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:02.852 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:02.852 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:02.852 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:02.852 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:02.852 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:02.852 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 10 client.?.objecter ms_handle_connect 0x7f87ec0a9ac0 2024-03-29T11:14:02.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:02.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 1 -- 172.21.15.170:0/1949894321 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1240900312 0 0) 0x7f87ec0725e0 con 0x7f87ec0b4100 2024-03-29T11:14:02.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 1 -- 172.21.15.170:0/1949894321 >> v1:172.21.15.170:6790/0 conn(0x7f87ec13ee90 legacy=0x7f87ec1deaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:02.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 1 -- 172.21.15.170:0/1949894321 >> v1:172.21.15.170:6789/0 conn(0x7f87ec0a9ac0 legacy=0x7f87ec139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:02.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:14:02.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 1 monclient: found mon.b 2024-03-29T11:14:02.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:02.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:02.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 1 -- 172.21.15.170:0/1949894321 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ec142520 con 0x7f87ec0b4100 2024-03-29T11:14:02.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 10 monclient: _finish_auth 0 2024-03-29T11:14:02.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:14:02.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:02.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87f1d32640 5 monclient: authenticate success, global_id 4300 2024-03-29T11:14:02.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87f1d32640 10 monclient: _renew_subs 2024-03-29T11:14:02.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87f1d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:02.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87f1d32640 1 -- 172.21.15.170:0/1949894321 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87ec06e7a0 con 0x7f87ec0b4100 2024-03-29T11:14:02.854 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87f1d32640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:02.854 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87f1d32640 10 monclient: _renew_subs 2024-03-29T11:14:02.854 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87f1d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:02.854 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87f1d32640 1 -- 172.21.15.170:0/1949894321 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87ec06eaa0 con 0x7f87ec0b4100 2024-03-29T11:14:02.854 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.873+0000 7f87f1d32640 1 librados: init done 2024-03-29T11:14:02.856 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.875+0000 7f87f1d32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:02.857 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.875+0000 7f87f1d32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:02.857 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.875+0000 7f87f1d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:02.857 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.875+0000 7f87f1d32640 1 -- 172.21.15.170:0/1949894321 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87ec075f10 con 0x7f87ec0b4100 2024-03-29T11:14:02.857 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.876+0000 7f87d3fff640 1 -- 172.21.15.170:0/1949894321 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87e0001c20 con 0x7f87ec0b4100 2024-03-29T11:14:02.857 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.876+0000 7f87d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:02.857 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.876+0000 7f87d3fff640 1 -- 172.21.15.170:0/1949894321 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f87e0004a00 con 0x7f87ec0b4100 2024-03-29T11:14:02.857 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.876+0000 7f87d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:02.857 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.876+0000 7f87d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:02.857 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.876+0000 7f87d3fff640 10 monclient: dump: 2024-03-29T11:14:02.857 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:02.857 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:02.857 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:02.857 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:02.858 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:02.858 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:02.858 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:02.858 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:02.858 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:02.858 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:02.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.878+0000 7f87d3fff640 1 -- 172.21.15.170:0/1949894321 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f87e001a9d0 con 0x7f87ec0b4100 2024-03-29T11:14:02.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.878+0000 7f87e9ffb640 4 set_mon_vals no callback set 2024-03-29T11:14:02.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.879+0000 7f87eb7fe640 10 monclient: get_auth_request con 0x7f87d806df70 auth_method 0 2024-03-29T11:14:02.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.879+0000 7f87d3fff640 10 client.4300.objecter ms_handle_connect 0x7f87d806df70 2024-03-29T11:14:02.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.879+0000 7f87d3fff640 1 -- 172.21.15.170:0/1949894321 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f87e00545e0 con 0x7f87ec0b4100 2024-03-29T11:14:02.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.879+0000 7f87d3fff640 10 client.4300.objecter ms_dispatch 0x7f87ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:02.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.879+0000 7f87d3fff640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:02.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.879+0000 7f87d3fff640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:02.859 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.879+0000 7f87d3fff640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-29T11:14:02.863 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.886+0000 7f87d3fff640 1 -- 172.21.15.170:0/1949894321 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87e0050880 con 0x7f87ec0b4100 2024-03-29T11:14:02.864 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.886+0000 7f87d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:02.864 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:02.886+0000 7f87d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:03.191 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.213+0000 7f87f1d32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:03.191 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.213+0000 7f87f1d32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:03.191 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.213+0000 7f87f1d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:03.191 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.214+0000 7f87f1d32640 1 -- 172.21.15.170:0/1949894321 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87ec0abb00 con 0x7f87ec0b4100 2024-03-29T11:14:03.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.217+0000 7f87d3fff640 1 -- 172.21.15.170:0/1949894321 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f87e0050a20 con 0x7f87ec0b4100 2024-03-29T11:14:03.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.217+0000 7f87d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:03.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.217+0000 7f87d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:03.195 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:14:03.195 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:03.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.221+0000 7f87f1d32640 10 librados: watch_flush enter 2024-03-29T11:14:03.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.222+0000 7f87f1d32640 10 librados: watch_flush exit 2024-03-29T11:14:03.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.222+0000 7f87f1d32640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-29T11:14:03.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.222+0000 7f87f1d32640 10 client.4300.objecter successfully canceled tick 2024-03-29T11:14:03.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.222+0000 7f87f1d32640 1 -- 172.21.15.170:0/1949894321 >> v1:172.21.15.170:6816/3298994444 conn(0x7f87d806df70 legacy=0x7f87d8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:03.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.222+0000 7f87f1d32640 10 monclient: shutdown 2024-03-29T11:14:03.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.222+0000 7f87f1d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:03.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.222+0000 7f87f1d32640 1 -- 172.21.15.170:0/1949894321 >> v1:172.21.15.184:6789/0 conn(0x7f87ec0b4100 legacy=0x7f87ec1d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:03.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.222+0000 7f87f1d32640 1 -- 172.21.15.170:0/1949894321 shutdown_connections 2024-03-29T11:14:03.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.223+0000 7f87f1d32640 1 -- 172.21.15.170:0/1949894321 >> 172.21.15.170:0/1949894321 conn(0x7f87ec0b35e0 msgr2=0x7f87ec0b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:03.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.225+0000 7f87f1d32640 1 -- 172.21.15.170:0/1949894321 shutdown_connections 2024-03-29T11:14:03.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.225+0000 7f87f1d32640 1 -- 172.21.15.170:0/1949894321 wait complete. 2024-03-29T11:14:03.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:03.225+0000 7f87f1d32640 1 librados: shutdown 2024-03-29T11:14:03.221 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:10.027 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:10.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.209+0000 7f557ffff640 10 monclient: get_monmap_and_config 2024-03-29T11:14:10.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.209+0000 7f557ffff640 10 monclient: build_initial_monmap 2024-03-29T11:14:10.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.209+0000 7f557ffff640 10 monclient: monmap: 2024-03-29T11:14:10.187 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:14:10.187 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:10.187 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:14:10.209958+0000 2024-03-29T11:14:10.188 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:14:10.209958+0000 2024-03-29T11:14:10.188 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:10.188 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:10.188 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:14:10.188 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:14:10.188 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:14:10.188 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:10.190 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.213+0000 7f557ffff640 1 Processor -- start 2024-03-29T11:14:10.192 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.214+0000 7f557ffff640 1 -- start start 2024-03-29T11:14:10.192 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.214+0000 7f557ffff640 10 monclient: init 2024-03-29T11:14:10.193 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.215+0000 7f557ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:10.193 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.215+0000 7f557ffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:14:10.193 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.215+0000 7f557ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f558014a400 addr v1:172.21.15.184:6789/0 2024-03-29T11:14:10.193 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.215+0000 7f557ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f55800aa500 addr v1:172.21.15.170:6790/0 2024-03-29T11:14:10.193 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.215+0000 7f557ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f5580149800 addr v1:172.21.15.170:6789/0 2024-03-29T11:14:10.193 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.215+0000 7f557ffff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55800aca50 con 0x7f5580149800 2024-03-29T11:14:10.193 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.215+0000 7f557ffff640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5580149cb0 con 0x7f558014a400 2024-03-29T11:14:10.193 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.215+0000 7f557ffff640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55800acc80 con 0x7f55800aa500 2024-03-29T11:14:10.193 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.215+0000 7f557ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:10.193 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.215+0000 7f557ffff640 10 monclient(hunting): authenticate will time out at 1031.526194s 2024-03-29T11:14:10.196 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.219+0000 7f557f7fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f5580149800 0x7f5580151ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:49495/0 (socket says 172.21.15.170:49495) 2024-03-29T11:14:10.196 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.219+0000 7f557f7fe640 1 -- 172.21.15.170:0/2540733884 learned_addr learned my addr 172.21.15.170:0/2540733884 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:14:10.201 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.222+0000 7f557dffb640 1 -- 172.21.15.170:0/2540733884 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294167275 0 0) 0x7f55800acc80 con 0x7f55800aa500 2024-03-29T11:14:10.201 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.222+0000 7f557dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:10.201 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.222+0000 7f557dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:10.201 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.222+0000 7f557dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:10.201 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.222+0000 7f557dffb640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:14:10.201 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.222+0000 7f557dffb640 1 -- 172.21.15.170:0/2540733884 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5568003520 con 0x7f55800aa500 2024-03-29T11:14:10.201 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.222+0000 7f557dffb640 1 -- 172.21.15.170:0/2540733884 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928318586 0 0) 0x7f5580149cb0 con 0x7f558014a400 2024-03-29T11:14:10.201 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.222+0000 7f557dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:10.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.222+0000 7f557dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:10.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.222+0000 7f557dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:10.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.222+0000 7f557dffb640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:14:10.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.222+0000 7f557dffb640 1 -- 172.21.15.170:0/2540733884 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55800acc80 con 0x7f558014a400 2024-03-29T11:14:10.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.223+0000 7f557dffb640 1 -- 172.21.15.170:0/2540733884 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775484921 0 0) 0x7f55800aca50 con 0x7f5580149800 2024-03-29T11:14:10.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.223+0000 7f557dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:10.206 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.223+0000 7f557dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:10.206 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.223+0000 7f557dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:10.206 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.223+0000 7f557dffb640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:14:10.206 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.223+0000 7f557dffb640 1 -- 172.21.15.170:0/2540733884 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5580149cb0 con 0x7f5580149800 2024-03-29T11:14:10.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.227+0000 7f557dffb640 1 -- 172.21.15.170:0/2540733884 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3206642071 0 0) 0x7f5580149cb0 con 0x7f5580149800 2024-03-29T11:14:10.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.227+0000 7f557dffb640 1 -- 172.21.15.170:0/2540733884 >> v1:172.21.15.170:6790/0 conn(0x7f55800aa500 legacy=0x7f55800ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:10.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.227+0000 7f557dffb640 1 -- 172.21.15.170:0/2540733884 >> v1:172.21.15.184:6789/0 conn(0x7f558014a400 legacy=0x7f55800aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:10.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.227+0000 7f557dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:14:10.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.227+0000 7f557dffb640 1 monclient: found mon.noname-a 2024-03-29T11:14:10.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.227+0000 7f557dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:10.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.227+0000 7f557dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:14:10.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.227+0000 7f557dffb640 1 -- 172.21.15.170:0/2540733884 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55800ac350 con 0x7f5580149800 2024-03-29T11:14:10.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.227+0000 7f557dffb640 10 monclient: _finish_auth 0 2024-03-29T11:14:10.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.227+0000 7f557dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:14:10.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.227+0000 7f557dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:10.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.227+0000 7f557dffb640 1 -- 172.21.15.170:0/2540733884 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f55700015b0 con 0x7f5580149800 2024-03-29T11:14:10.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.227+0000 7f557dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:10.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.227+0000 7f557dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:10.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.227+0000 7f557dffb640 10 monclient: dump: 2024-03-29T11:14:10.209 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:10.210 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:10.210 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:10.210 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:10.210 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:10.210 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:10.210 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:10.210 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:10.211 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:10.211 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:10.211 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.228+0000 7f557ffff640 5 monclient: authenticate success, global_id 4308 2024-03-29T11:14:10.211 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.228+0000 7f557ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:10.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.231+0000 7f557dffb640 1 -- 172.21.15.170:0/2540733884 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5570001710 con 0x7f5580149800 2024-03-29T11:14:10.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.231+0000 7f557dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:10.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.231+0000 7f557dffb640 1 -- 172.21.15.170:0/2540733884 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5570001b20 con 0x7f5580149800 2024-03-29T11:14:10.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.231+0000 7f557dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:10.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.231+0000 7f557dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:10.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.231+0000 7f557dffb640 10 monclient: dump: 2024-03-29T11:14:10.212 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:10.212 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:10.212 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:10.213 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:10.213 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:10.213 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:10.213 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:10.213 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:10.213 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:10.213 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:10.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.231+0000 7f557ffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:10.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.231+0000 7f557ffff640 4 set_mon_vals no callback set 2024-03-29T11:14:10.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.232+0000 7f557ffff640 10 monclient: shutdown 2024-03-29T11:14:10.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.232+0000 7f557ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:10.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.232+0000 7f557ffff640 1 -- 172.21.15.170:0/2540733884 >> v1:172.21.15.170:6789/0 conn(0x7f5580149800 legacy=0x7f5580151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:10.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.232+0000 7f557ffff640 1 -- 172.21.15.170:0/2540733884 shutdown_connections 2024-03-29T11:14:10.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.232+0000 7f557ffff640 1 -- 172.21.15.170:0/2540733884 >> 172.21.15.170:0/2540733884 conn(0x7f55800b3a30 msgr2=0x7f55800b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:10.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.233+0000 7f557ffff640 1 -- 172.21.15.170:0/2540733884 shutdown_connections 2024-03-29T11:14:10.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.233+0000 7f557ffff640 1 -- 172.21.15.170:0/2540733884 wait complete. 2024-03-29T11:14:10.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.234+0000 7f557ffff640 10 monclient: build_initial_monmap 2024-03-29T11:14:10.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.234+0000 7f557ffff640 10 monclient: monmap: 2024-03-29T11:14:10.214 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:14:10.214 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:10.214 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:14:10.214 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:14:10.214 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:10.214 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:10.214 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:14:10.214 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:14:10.214 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:14:10.214 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:10.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.234+0000 7f557ffff640 1 librados: starting msgr at 2024-03-29T11:14:10.216 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.234+0000 7f557ffff640 1 librados: starting objecter 2024-03-29T11:14:10.216 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.235+0000 7f557ffff640 1 Processor -- start 2024-03-29T11:14:10.217 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.235+0000 7f557ffff640 1 -- start start 2024-03-29T11:14:10.217 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.235+0000 7f557ffff640 1 librados: setting wanted keys 2024-03-29T11:14:10.217 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.235+0000 7f557ffff640 1 librados: calling monclient init 2024-03-29T11:14:10.217 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.235+0000 7f557ffff640 10 monclient: init 2024-03-29T11:14:10.217 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.236+0000 7f557ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:10.217 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.236+0000 7f557ffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:14:10.217 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.237+0000 7f557ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f55800aa500 addr v1:172.21.15.184:6789/0 2024-03-29T11:14:10.217 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.237+0000 7f557ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f5580149800 addr v1:172.21.15.170:6789/0 2024-03-29T11:14:10.217 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.237+0000 7f557ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f558014a400 addr v1:172.21.15.170:6790/0 2024-03-29T11:14:10.217 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.237+0000 7f557ffff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55801df0a0 con 0x7f5580149800 2024-03-29T11:14:10.217 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.237+0000 7f557ffff640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558013ee30 con 0x7f55800aa500 2024-03-29T11:14:10.217 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.237+0000 7f557ffff640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55801df2d0 con 0x7f558014a400 2024-03-29T11:14:10.217 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.237+0000 7f557ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:10.217 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.237+0000 7f557ffff640 10 monclient(hunting): authenticate will time out at 1031.548222s 2024-03-29T11:14:10.218 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.239+0000 7f557effd640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f55800aa500 0x7f55801de990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:48561/0 (socket says 172.21.15.170:48561) 2024-03-29T11:14:10.218 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.239+0000 7f557effd640 1 -- 172.21.15.170:0/1642486145 learned_addr learned my addr 172.21.15.170:0/1642486145 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:14:10.218 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.240+0000 7f55677fe640 10 client.?.objecter ms_handle_connect 0x7f55800aa500 2024-03-29T11:14:10.218 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.240+0000 7f55677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:10.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.242+0000 7f55677fe640 1 -- 172.21.15.170:0/1642486145 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814481083 0 0) 0x7f558013ee30 con 0x7f55800aa500 2024-03-29T11:14:10.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.242+0000 7f55677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:10.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.242+0000 7f55677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:10.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.242+0000 7f55677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:10.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.242+0000 7f55677fe640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:14:10.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.242+0000 7f55677fe640 1 -- 172.21.15.170:0/1642486145 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f556c004eb0 con 0x7f55800aa500 2024-03-29T11:14:10.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.243+0000 7f55677fe640 10 client.?.objecter ms_handle_connect 0x7f5580149800 2024-03-29T11:14:10.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.243+0000 7f55677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:10.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.243+0000 7f55677fe640 1 -- 172.21.15.170:0/1642486145 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062764508 0 0) 0x7f55801df0a0 con 0x7f5580149800 2024-03-29T11:14:10.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.243+0000 7f55677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:10.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.243+0000 7f55677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:10.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.243+0000 7f55677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:10.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.243+0000 7f55677fe640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:14:10.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.243+0000 7f55677fe640 1 -- 172.21.15.170:0/1642486145 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f558013ee30 con 0x7f5580149800 2024-03-29T11:14:10.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 1 -- 172.21.15.170:0/1642486145 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3436084257 0 0) 0x7f556c004eb0 con 0x7f55800aa500 2024-03-29T11:14:10.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 1 -- 172.21.15.170:0/1642486145 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f556c005490 con 0x7f55800aa500 2024-03-29T11:14:10.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 1 -- 172.21.15.170:0/1642486145 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5578001c70 con 0x7f55800aa500 2024-03-29T11:14:10.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:10.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:10.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 10 monclient(hunting): dump: 2024-03-29T11:14:10.225 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:10.225 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:10.225 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:10.225 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:10.225 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:10.225 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:10.225 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:10.225 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:10.226 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:10.226 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:10.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:10.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 1 -- 172.21.15.170:0/1642486145 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1481779469 0 0) 0x7f556c005490 con 0x7f55800aa500 2024-03-29T11:14:10.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 1 -- 172.21.15.170:0/1642486145 >> v1:172.21.15.170:6790/0 conn(0x7f558014a400 legacy=0x7f5580139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:10.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 1 -- 172.21.15.170:0/1642486145 >> v1:172.21.15.170:6789/0 conn(0x7f5580149800 legacy=0x7f5580138a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:10.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 10 monclient: _finish_hunting 0 2024-03-29T11:14:10.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 1 monclient: found mon.b 2024-03-29T11:14:10.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:10.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:10.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 1 -- 172.21.15.170:0/1642486145 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5580146310 con 0x7f55800aa500 2024-03-29T11:14:10.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 10 monclient: _finish_auth 0 2024-03-29T11:14:10.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 10 monclient: _check_auth_tickets 2024-03-29T11:14:10.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:10.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f557ffff640 5 monclient: authenticate success, global_id 4306 2024-03-29T11:14:10.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f557ffff640 10 monclient: _renew_subs 2024-03-29T11:14:10.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f557ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:10.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f557ffff640 1 -- 172.21.15.170:0/1642486145 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55801465c0 con 0x7f55800aa500 2024-03-29T11:14:10.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f557ffff640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:10.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 1 -- 172.21.15.170:0/1642486145 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5578003990 con 0x7f55800aa500 2024-03-29T11:14:10.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.244+0000 7f55677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:10.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.245+0000 7f55677fe640 1 -- 172.21.15.170:0/1642486145 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5578003d60 con 0x7f55800aa500 2024-03-29T11:14:10.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.245+0000 7f55677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:10.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.245+0000 7f55677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:10.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.245+0000 7f55677fe640 10 monclient: dump: 2024-03-29T11:14:10.229 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:10.229 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:10.229 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:10.229 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:10.229 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:10.229 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:10.229 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:10.229 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:10.230 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:10.230 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:10.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.245+0000 7f557ffff640 10 monclient: _renew_subs 2024-03-29T11:14:10.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.245+0000 7f557ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:10.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.245+0000 7f557ffff640 1 -- 172.21.15.170:0/1642486145 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55801e6e50 con 0x7f55800aa500 2024-03-29T11:14:10.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.245+0000 7f557ffff640 1 librados: init done 2024-03-29T11:14:10.233 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.247+0000 7f557d7fa640 4 set_mon_vals no callback set 2024-03-29T11:14:10.233 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.248+0000 7f55677fe640 1 -- 172.21.15.170:0/1642486145 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f5578003f40 con 0x7f55800aa500 2024-03-29T11:14:10.233 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.249+0000 7f55677fe640 1 -- 172.21.15.170:0/1642486145 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f5578084df0 con 0x7f55800aa500 2024-03-29T11:14:10.233 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.249+0000 7f55677fe640 10 client.4306.objecter ms_dispatch 0x7f5580000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:10.233 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.249+0000 7f55677fe640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:10.233 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.249+0000 7f55677fe640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:10.234 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.249+0000 7f55677fe640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-29T11:14:10.234 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.249+0000 7f557e7fc640 10 monclient: get_auth_request con 0x7f556c06df50 auth_method 0 2024-03-29T11:14:10.234 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.250+0000 7f55677fe640 10 client.4306.objecter ms_handle_connect 0x7f556c06df50 2024-03-29T11:14:10.234 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.250+0000 7f557ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:10.234 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.250+0000 7f557ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:10.234 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.250+0000 7f557ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:10.234 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.250+0000 7f557ffff640 1 -- 172.21.15.170:0/1642486145 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5560005080 con 0x7f55800aa500 2024-03-29T11:14:10.238 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.258+0000 7f55677fe640 1 -- 172.21.15.170:0/1642486145 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f55780506d0 con 0x7f55800aa500 2024-03-29T11:14:10.238 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.258+0000 7f55677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:10.238 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.258+0000 7f55677fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:10.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.580+0000 7f557ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:10.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.580+0000 7f557ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:10.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.580+0000 7f557ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:10.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.580+0000 7f557ffff640 1 -- 172.21.15.170:0/1642486145 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5560005330 con 0x7f55800aa500 2024-03-29T11:14:10.562 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:14:10.562 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:10.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.582+0000 7f55677fe640 1 -- 172.21.15.170:0/1642486145 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f5578052600 con 0x7f55800aa500 2024-03-29T11:14:10.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.582+0000 7f55677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:10.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.582+0000 7f55677fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:10.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.585+0000 7f557ffff640 10 librados: watch_flush enter 2024-03-29T11:14:10.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.585+0000 7f557ffff640 10 librados: watch_flush exit 2024-03-29T11:14:10.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.585+0000 7f557ffff640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-29T11:14:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.585+0000 7f557ffff640 10 client.4306.objecter successfully canceled tick 2024-03-29T11:14:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.585+0000 7f557ffff640 1 -- 172.21.15.170:0/1642486145 >> v1:172.21.15.170:6816/3298994444 conn(0x7f556c06df50 legacy=0x7f556c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.585+0000 7f557ffff640 10 monclient: shutdown 2024-03-29T11:14:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.585+0000 7f557ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.585+0000 7f557ffff640 1 -- 172.21.15.170:0/1642486145 >> v1:172.21.15.184:6789/0 conn(0x7f55800aa500 legacy=0x7f55801de990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:10.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.588+0000 7f557ffff640 1 -- 172.21.15.170:0/1642486145 shutdown_connections 2024-03-29T11:14:10.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.588+0000 7f557ffff640 1 -- 172.21.15.170:0/1642486145 >> 172.21.15.170:0/1642486145 conn(0x7f55800b3a30 msgr2=0x7f5580069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:10.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.589+0000 7f557ffff640 1 -- 172.21.15.170:0/1642486145 shutdown_connections 2024-03-29T11:14:10.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.589+0000 7f557ffff640 1 -- 172.21.15.170:0/1642486145 wait complete. 2024-03-29T11:14:10.567 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:10.590+0000 7f557ffff640 1 librados: shutdown 2024-03-29T11:14:10.587 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:17.397 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:17.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.577+0000 7fc101bab640 10 monclient: get_monmap_and_config 2024-03-29T11:14:17.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.577+0000 7fc101bab640 10 monclient: build_initial_monmap 2024-03-29T11:14:17.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.577+0000 7fc101bab640 10 monclient: monmap: 2024-03-29T11:14:17.555 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:14:17.555 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:17.555 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:14:17.578728+0000 2024-03-29T11:14:17.555 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:14:17.578728+0000 2024-03-29T11:14:17.555 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:17.556 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:17.556 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:14:17.556 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:14:17.556 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:14:17.556 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:17.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.578+0000 7fc101bab640 1 Processor -- start 2024-03-29T11:14:17.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.578+0000 7fc101bab640 1 -- start start 2024-03-29T11:14:17.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.578+0000 7fc101bab640 10 monclient: init 2024-03-29T11:14:17.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.579+0000 7fc101bab640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:17.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.579+0000 7fc101bab640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:17.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.579+0000 7fc101bab640 10 monclient(hunting): picked mon.noname-b con 0x7fc0f4037230 addr v1:172.21.15.170:6790/0 2024-03-29T11:14:17.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.579+0000 7fc101bab640 10 monclient(hunting): picked mon.noname-a con 0x7fc0f4033a80 addr v1:172.21.15.170:6789/0 2024-03-29T11:14:17.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.579+0000 7fc101bab640 10 monclient(hunting): picked mon.noname-c con 0x7fc0f40d4770 addr v1:172.21.15.184:6789/0 2024-03-29T11:14:17.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.579+0000 7fc101bab640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f4035bf0 con 0x7fc0f4033a80 2024-03-29T11:14:17.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.579+0000 7fc101bab640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f402c150 con 0x7fc0f40d4770 2024-03-29T11:14:17.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.579+0000 7fc101bab640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f40356f0 con 0x7fc0f4037230 2024-03-29T11:14:17.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.579+0000 7fc101bab640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:17.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.579+0000 7fc101bab640 10 monclient(hunting): authenticate will time out at 1038.891968s 2024-03-29T11:14:17.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.580+0000 7fc1013aa640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fc0f40d4770 0x7fc0f40d6b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:45084/0 (socket says 172.21.15.170:45084) 2024-03-29T11:14:17.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.580+0000 7fc1013aa640 1 -- 172.21.15.170:0/1918200316 learned_addr learned my addr 172.21.15.170:0/1918200316 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:14:17.560 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.580+0000 7fc0fb7fe640 1 -- 172.21.15.170:0/1918200316 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662008518 0 0) 0x7fc0f402c150 con 0x7fc0f40d4770 2024-03-29T11:14:17.560 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.580+0000 7fc0fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:17.560 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.580+0000 7fc0fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:17.560 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.580+0000 7fc0fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:17.560 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.580+0000 7fc0fb7fe640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:14:17.560 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.580+0000 7fc0fb7fe640 1 -- 172.21.15.170:0/1918200316 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0ec003520 con 0x7fc0f40d4770 2024-03-29T11:14:17.560 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 1 -- 172.21.15.170:0/1918200316 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579419192 0 0) 0x7fc0f4035bf0 con 0x7fc0f4033a80 2024-03-29T11:14:17.560 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:17.560 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:17.560 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:17.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:14:17.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 1 -- 172.21.15.170:0/1918200316 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0f402c150 con 0x7fc0f4033a80 2024-03-29T11:14:17.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 1 -- 172.21.15.170:0/1918200316 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3128534032 0 0) 0x7fc0ec003520 con 0x7fc0f40d4770 2024-03-29T11:14:17.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 1 -- 172.21.15.170:0/1918200316 >> v1:172.21.15.170:6790/0 conn(0x7fc0f4037230 legacy=0x7fc0f4036080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:17.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 1 -- 172.21.15.170:0/1918200316 >> v1:172.21.15.170:6789/0 conn(0x7fc0f4033a80 legacy=0x7fc0f40cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:17.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:14:17.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 1 monclient: found mon.noname-c 2024-03-29T11:14:17.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:17.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:14:17.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 1 -- 172.21.15.170:0/1918200316 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f4035880 con 0x7fc0f40d4770 2024-03-29T11:14:17.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:14:17.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:14:17.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:17.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 1 -- 172.21.15.170:0/1918200316 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fc0fc04f730 con 0x7fc0f40d4770 2024-03-29T11:14:17.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:17.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:17.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc0fb7fe640 10 monclient: dump: 2024-03-29T11:14:17.562 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:17.562 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:17.562 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:17.562 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:17.562 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:17.562 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:17.562 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:17.562 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:17.562 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:17.562 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:17.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc101bab640 5 monclient: authenticate success, global_id 4309 2024-03-29T11:14:17.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.581+0000 7fc101bab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:17.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.582+0000 7fc0fb7fe640 1 -- 172.21.15.170:0/1918200316 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0fc075f10 con 0x7fc0f40d4770 2024-03-29T11:14:17.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.582+0000 7fc0fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:17.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.582+0000 7fc101bab640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:17.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.582+0000 7fc101bab640 4 set_mon_vals no callback set 2024-03-29T11:14:17.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.582+0000 7fc0fb7fe640 1 -- 172.21.15.170:0/1918200316 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fc0fc0758b0 con 0x7fc0f40d4770 2024-03-29T11:14:17.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.582+0000 7fc0fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:17.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.582+0000 7fc0fb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:17.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.582+0000 7fc0fb7fe640 10 monclient: dump: 2024-03-29T11:14:17.563 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:17.564 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:17.564 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:17.564 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:17.564 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:17.564 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:17.564 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:17.564 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:17.564 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:17.564 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:17.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.582+0000 7fc101bab640 10 monclient: shutdown 2024-03-29T11:14:17.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.582+0000 7fc101bab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:17.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.582+0000 7fc101bab640 1 -- 172.21.15.170:0/1918200316 >> v1:172.21.15.184:6789/0 conn(0x7fc0f40d4770 legacy=0x7fc0f40d6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:17.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.582+0000 7fc101bab640 1 -- 172.21.15.170:0/1918200316 shutdown_connections 2024-03-29T11:14:17.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.582+0000 7fc101bab640 1 -- 172.21.15.170:0/1918200316 >> 172.21.15.170:0/1918200316 conn(0x7fc0f403a590 msgr2=0x7fc0f40367e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:17.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.583+0000 7fc101bab640 1 -- 172.21.15.170:0/1918200316 shutdown_connections 2024-03-29T11:14:17.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.583+0000 7fc101bab640 1 -- 172.21.15.170:0/1918200316 wait complete. 2024-03-29T11:14:17.572 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.584+0000 7fc101bab640 10 monclient: build_initial_monmap 2024-03-29T11:14:17.572 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.584+0000 7fc101bab640 10 monclient: monmap: 2024-03-29T11:14:17.572 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:14:17.572 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:17.572 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:14:17.572 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:14:17.572 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:17.572 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:17.572 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:14:17.572 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:14:17.572 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:14:17.572 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:17.572 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.584+0000 7fc101bab640 1 librados: starting msgr at 2024-03-29T11:14:17.572 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.584+0000 7fc101bab640 1 librados: starting objecter 2024-03-29T11:14:17.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.584+0000 7fc101bab640 1 Processor -- start 2024-03-29T11:14:17.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.584+0000 7fc101bab640 1 -- start start 2024-03-29T11:14:17.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.584+0000 7fc101bab640 1 librados: setting wanted keys 2024-03-29T11:14:17.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.584+0000 7fc101bab640 1 librados: calling monclient init 2024-03-29T11:14:17.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.584+0000 7fc101bab640 10 monclient: init 2024-03-29T11:14:17.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.585+0000 7fc101bab640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:17.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.585+0000 7fc101bab640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:17.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.585+0000 7fc101bab640 10 monclient(hunting): picked mon.noname-a con 0x7fc0f4033a80 addr v1:172.21.15.170:6789/0 2024-03-29T11:14:17.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.585+0000 7fc101bab640 10 monclient(hunting): picked mon.noname-b con 0x7fc0f4037230 addr v1:172.21.15.184:6789/0 2024-03-29T11:14:17.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.586+0000 7fc101bab640 10 monclient(hunting): picked mon.noname-c con 0x7fc0f40d4770 addr v1:172.21.15.170:6790/0 2024-03-29T11:14:17.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.586+0000 7fc101bab640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f40d5cd0 con 0x7fc0f4033a80 2024-03-29T11:14:17.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.586+0000 7fc101bab640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f40d5fb0 con 0x7fc0f4037230 2024-03-29T11:14:17.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.586+0000 7fc101bab640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f40d6290 con 0x7fc0f40d4770 2024-03-29T11:14:17.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.586+0000 7fc101bab640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:17.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.586+0000 7fc101bab640 10 monclient(hunting): authenticate will time out at 1038.898087s 2024-03-29T11:14:17.574 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.586+0000 7fc0fbfff640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fc0f4037230 0x7fc0f40d6a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:45300/0 (socket says 172.21.15.170:45300) 2024-03-29T11:14:17.574 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.586+0000 7fc0fbfff640 1 -- 172.21.15.170:0/1061149007 learned_addr learned my addr 172.21.15.170:0/1061149007 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:14:17.574 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.586+0000 7fc0f97fa640 10 client.?.objecter ms_handle_connect 0x7fc0f4037230 2024-03-29T11:14:17.574 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.586+0000 7fc0f97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:17.574 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.586+0000 7fc0f97fa640 10 client.?.objecter ms_handle_connect 0x7fc0f40d4770 2024-03-29T11:14:17.574 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.586+0000 7fc0f97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:17.574 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.586+0000 7fc0f97fa640 1 -- 172.21.15.170:0/1061149007 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227139965 0 0) 0x7fc0f40d5fb0 con 0x7fc0f4037230 2024-03-29T11:14:17.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.586+0000 7fc0f97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:14:17.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.586+0000 7fc0f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:17.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.586+0000 7fc0f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:17.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.586+0000 7fc0f97fa640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:14:17.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.586+0000 7fc0f97fa640 1 -- 172.21.15.170:0/1061149007 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0e0004eb0 con 0x7fc0f4037230 2024-03-29T11:14:17.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 10 client.?.objecter ms_handle_connect 0x7fc0f4033a80 2024-03-29T11:14:17.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:17.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 1 -- 172.21.15.170:0/1061149007 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784050180 0 0) 0x7fc0f40d6290 con 0x7fc0f40d4770 2024-03-29T11:14:17.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:14:17.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:17.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:17.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:14:17.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 1 -- 172.21.15.170:0/1061149007 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0f40d5fb0 con 0x7fc0f40d4770 2024-03-29T11:14:17.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 1 -- 172.21.15.170:0/1061149007 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1593880970 0 0) 0x7fc0e0004eb0 con 0x7fc0f4037230 2024-03-29T11:14:17.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 1 -- 172.21.15.170:0/1061149007 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0f40d6290 con 0x7fc0f4037230 2024-03-29T11:14:17.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 1 -- 172.21.15.170:0/1061149007 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fc0e8001a10 con 0x7fc0f4037230 2024-03-29T11:14:17.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:17.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:17.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 10 monclient(hunting): dump: 2024-03-29T11:14:17.576 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:17.576 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:17.576 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:17.576 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:17.576 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:17.576 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:17.576 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:17.576 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:17.576 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:17.576 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:17.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:17.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 1 -- 172.21.15.170:0/1061149007 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007233805 0 0) 0x7fc0f40d5cd0 con 0x7fc0f4033a80 2024-03-29T11:14:17.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:14:17.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:17.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:17.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:14:17.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 1 -- 172.21.15.170:0/1061149007 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0e0004eb0 con 0x7fc0f4033a80 2024-03-29T11:14:17.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 1 -- 172.21.15.170:0/1061149007 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1591774163 0 0) 0x7fc0f40d6290 con 0x7fc0f4037230 2024-03-29T11:14:17.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 1 -- 172.21.15.170:0/1061149007 >> v1:172.21.15.170:6790/0 conn(0x7fc0f40d4770 legacy=0x7fc0f416cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:17.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 1 -- 172.21.15.170:0/1061149007 >> v1:172.21.15.170:6789/0 conn(0x7fc0f4033a80 legacy=0x7fc0f40bd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:17.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:14:17.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 1 monclient: found mon.b 2024-03-29T11:14:17.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:17.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:17.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.587+0000 7fc0f97fa640 1 -- 172.21.15.170:0/1061149007 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f40cc8e0 con 0x7fc0f4037230 2024-03-29T11:14:17.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.588+0000 7fc0f97fa640 10 monclient: _finish_auth 0 2024-03-29T11:14:17.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.588+0000 7fc0f97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:14:17.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.588+0000 7fc0f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:17.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.588+0000 7fc101bab640 5 monclient: authenticate success, global_id 4312 2024-03-29T11:14:17.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.588+0000 7fc101bab640 10 monclient: _renew_subs 2024-03-29T11:14:17.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.588+0000 7fc101bab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:17.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.588+0000 7fc101bab640 1 -- 172.21.15.170:0/1061149007 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f40be070 con 0x7fc0f4037230 2024-03-29T11:14:17.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.588+0000 7fc101bab640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:17.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.588+0000 7fc101bab640 10 monclient: _renew_subs 2024-03-29T11:14:17.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.588+0000 7fc101bab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:17.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.588+0000 7fc101bab640 1 -- 172.21.15.170:0/1061149007 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f40d7d40 con 0x7fc0f4037230 2024-03-29T11:14:17.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.588+0000 7fc101bab640 1 librados: init done 2024-03-29T11:14:17.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.588+0000 7fc0f97fa640 1 -- 172.21.15.170:0/1061149007 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0e8001cd0 con 0x7fc0f4037230 2024-03-29T11:14:17.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.588+0000 7fc0f97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:17.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.588+0000 7fc0f97fa640 1 -- 172.21.15.170:0/1061149007 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fc0e8003a60 con 0x7fc0f4037230 2024-03-29T11:14:17.579 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.588+0000 7fc0faffd640 4 set_mon_vals no callback set 2024-03-29T11:14:17.579 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.588+0000 7fc0f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:17.579 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.588+0000 7fc0f97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:17.579 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.588+0000 7fc0f97fa640 10 monclient: dump: 2024-03-29T11:14:17.579 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:17.579 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:17.579 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:17.579 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:17.579 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:17.579 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:17.579 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:17.579 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:17.579 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:17.579 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:17.579 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.590+0000 7fc101bab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:17.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.590+0000 7fc101bab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:17.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.590+0000 7fc101bab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:17.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.590+0000 7fc101bab640 1 -- 172.21.15.170:0/1061149007 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0dc005080 con 0x7fc0f4037230 2024-03-29T11:14:17.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.590+0000 7fc0f97fa640 1 -- 172.21.15.170:0/1061149007 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fc0e8003bc0 con 0x7fc0f4037230 2024-03-29T11:14:17.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.591+0000 7fc0f97fa640 1 -- 172.21.15.170:0/1061149007 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fc0e8084b00 con 0x7fc0f4037230 2024-03-29T11:14:17.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.591+0000 7fc0f97fa640 10 client.4312.objecter ms_dispatch 0x7fc0fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:17.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.591+0000 7fc0f97fa640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:17.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.591+0000 7fc0f97fa640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:17.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.591+0000 7fc100ba9640 10 monclient: get_auth_request con 0x7fc0e006db10 auth_method 0 2024-03-29T11:14:17.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.591+0000 7fc0f97fa640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-29T11:14:17.581 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.592+0000 7fc0f97fa640 10 client.4312.objecter ms_handle_connect 0x7fc0e006db10 2024-03-29T11:14:17.581 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.598+0000 7fc0f97fa640 1 -- 172.21.15.170:0/1061149007 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc0e80503e0 con 0x7fc0f4037230 2024-03-29T11:14:17.581 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.598+0000 7fc0f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:17.581 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:17.598+0000 7fc0f97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:17.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.017+0000 7fc101bab640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:17.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.017+0000 7fc101bab640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:17.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.017+0000 7fc101bab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:17.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.017+0000 7fc101bab640 1 -- 172.21.15.170:0/1061149007 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0dc005330 con 0x7fc0f4037230 2024-03-29T11:14:17.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.018+0000 7fc0f97fa640 1 -- 172.21.15.170:0/1061149007 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fc0e8052310 con 0x7fc0f4037230 2024-03-29T11:14:17.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.018+0000 7fc0f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:17.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.018+0000 7fc0f97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:17.996 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:14:17.996 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:18.000 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.022+0000 7fc101bab640 10 librados: watch_flush enter 2024-03-29T11:14:18.000 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.022+0000 7fc101bab640 10 librados: watch_flush exit 2024-03-29T11:14:18.000 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.022+0000 7fc101bab640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-29T11:14:18.000 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.022+0000 7fc101bab640 10 client.4312.objecter successfully canceled tick 2024-03-29T11:14:18.000 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.022+0000 7fc101bab640 1 -- 172.21.15.170:0/1061149007 >> v1:172.21.15.170:6816/3298994444 conn(0x7fc0e006db10 legacy=0x7fc0e006ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:18.000 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.022+0000 7fc101bab640 10 monclient: shutdown 2024-03-29T11:14:18.001 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.022+0000 7fc101bab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:18.001 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.022+0000 7fc101bab640 1 -- 172.21.15.170:0/1061149007 >> v1:172.21.15.184:6789/0 conn(0x7fc0f4037230 legacy=0x7fc0f40d6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:18.001 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.023+0000 7fc101bab640 1 -- 172.21.15.170:0/1061149007 shutdown_connections 2024-03-29T11:14:18.001 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.023+0000 7fc101bab640 1 -- 172.21.15.170:0/1061149007 >> 172.21.15.170:0/1061149007 conn(0x7fc0f403a590 msgr2=0x7fc0f40c07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:18.001 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.023+0000 7fc101bab640 1 -- 172.21.15.170:0/1061149007 shutdown_connections 2024-03-29T11:14:18.001 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.023+0000 7fc101bab640 1 -- 172.21.15.170:0/1061149007 wait complete. 2024-03-29T11:14:18.001 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:18.023+0000 7fc101bab640 1 librados: shutdown 2024-03-29T11:14:18.018 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:24.824 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:24.974 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.996+0000 7fbdaae5c640 10 monclient: get_monmap_and_config 2024-03-29T11:14:24.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.996+0000 7fbdaae5c640 10 monclient: build_initial_monmap 2024-03-29T11:14:24.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.996+0000 7fbdaae5c640 10 monclient: monmap: 2024-03-29T11:14:24.975 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:14:24.975 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:24.975 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:14:24.997963+0000 2024-03-29T11:14:24.975 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:14:24.997963+0000 2024-03-29T11:14:24.975 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:24.975 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:24.975 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:14:24.975 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:14:24.975 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:14:24.975 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:24.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.997+0000 7fbdaae5c640 1 Processor -- start 2024-03-29T11:14:24.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.997+0000 7fbdaae5c640 1 -- start start 2024-03-29T11:14:24.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.997+0000 7fbdaae5c640 10 monclient: init 2024-03-29T11:14:24.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.998+0000 7fbdaae5c640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:24.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.998+0000 7fbdaae5c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:24.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.998+0000 7fbdaae5c640 10 monclient(hunting): picked mon.noname-b con 0x7fbda41457f0 addr v1:172.21.15.170:6790/0 2024-03-29T11:14:24.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.998+0000 7fbdaae5c640 10 monclient(hunting): picked mon.noname-a con 0x7fbda413f650 addr v1:172.21.15.170:6789/0 2024-03-29T11:14:24.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.998+0000 7fbdaae5c640 10 monclient(hunting): picked mon.noname-c con 0x7fbda40a9d70 addr v1:172.21.15.184:6789/0 2024-03-29T11:14:24.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.998+0000 7fbdaae5c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda40aaef0 con 0x7fbda413f650 2024-03-29T11:14:24.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.998+0000 7fbda8bf8640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fbda41457f0 0x7fbda413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:39506/0 (socket says 172.21.15.170:39506) 2024-03-29T11:14:24.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.998+0000 7fbda8bf8640 1 -- 172.21.15.170:0/1338031296 learned_addr learned my addr 172.21.15.170:0/1338031296 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:14:24.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.998+0000 7fbdaae5c640 1 -- 172.21.15.170:0/1338031296 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda4145bd0 con 0x7fbda40a9d70 2024-03-29T11:14:24.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.999+0000 7fbdaae5c640 1 -- 172.21.15.170:0/1338031296 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda4145eb0 con 0x7fbda41457f0 2024-03-29T11:14:24.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.999+0000 7fbdaae5c640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:24.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.999+0000 7fbdaae5c640 10 monclient(hunting): authenticate will time out at 1046.311099s 2024-03-29T11:14:24.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.999+0000 7fbda37fe640 1 -- 172.21.15.170:0/1338031296 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184432427 0 0) 0x7fbda4145eb0 con 0x7fbda41457f0 2024-03-29T11:14:24.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.999+0000 7fbda37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:24.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.999+0000 7fbda37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:24.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.999+0000 7fbda37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:24.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.999+0000 7fbda37fe640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:14:24.979 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.999+0000 7fbda37fe640 1 -- 172.21.15.170:0/1338031296 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd94003520 con 0x7fbda41457f0 2024-03-29T11:14:24.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.999+0000 7fbda37fe640 1 -- 172.21.15.170:0/1338031296 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513944794 0 0) 0x7fbda40aaef0 con 0x7fbda413f650 2024-03-29T11:14:24.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.999+0000 7fbda37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:24.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.999+0000 7fbda37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:24.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.999+0000 7fbda37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:24.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.999+0000 7fbda37fe640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:14:24.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:24.999+0000 7fbda37fe640 1 -- 172.21.15.170:0/1338031296 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbda4145eb0 con 0x7fbda413f650 2024-03-29T11:14:24.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 1 -- 172.21.15.170:0/1338031296 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 319402206 0 0) 0x7fbd94003520 con 0x7fbda41457f0 2024-03-29T11:14:24.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 1 -- 172.21.15.170:0/1338031296 >> v1:172.21.15.184:6789/0 conn(0x7fbda40a9d70 legacy=0x7fbda414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:24.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 1 -- 172.21.15.170:0/1338031296 >> v1:172.21.15.170:6789/0 conn(0x7fbda413f650 legacy=0x7fbda40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:24.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:14:24.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 1 monclient: found mon.noname-b 2024-03-29T11:14:24.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:24.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:14:24.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 1 -- 172.21.15.170:0/1338031296 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda4146340 con 0x7fbda41457f0 2024-03-29T11:14:24.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 10 monclient: _finish_auth 0 2024-03-29T11:14:24.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:14:24.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:24.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 1 -- 172.21.15.170:0/1338031296 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fbd98001610 con 0x7fbda41457f0 2024-03-29T11:14:24.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:24.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:24.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 10 monclient: dump: 2024-03-29T11:14:24.981 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:24.982 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:24.982 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:24.982 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:24.982 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:24.982 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:24.982 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:24.982 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:24.982 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:24.982 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:24.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbdaae5c640 5 monclient: authenticate success, global_id 4334 2024-03-29T11:14:24.983 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbdaae5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:24.983 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 1 -- 172.21.15.170:0/1338031296 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd98001770 con 0x7fbda41457f0 2024-03-29T11:14:24.983 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:24.983 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbdaae5c640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:24.983 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 1 -- 172.21.15.170:0/1338031296 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fbd98001bc0 con 0x7fbda41457f0 2024-03-29T11:14:24.983 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:24.986 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbdaae5c640 4 set_mon_vals no callback set 2024-03-29T11:14:24.986 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:24.986 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.000+0000 7fbda37fe640 10 monclient: dump: 2024-03-29T11:14:24.986 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:24.986 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:24.986 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:24.986 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:24.986 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:24.986 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:24.986 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:24.986 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:24.986 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:24.986 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:24.987 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.001+0000 7fbdaae5c640 10 monclient: shutdown 2024-03-29T11:14:24.987 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.001+0000 7fbdaae5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:24.987 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.001+0000 7fbdaae5c640 1 -- 172.21.15.170:0/1338031296 >> v1:172.21.15.170:6790/0 conn(0x7fbda41457f0 legacy=0x7fbda413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:24.987 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.001+0000 7fbdaae5c640 1 -- 172.21.15.170:0/1338031296 shutdown_connections 2024-03-29T11:14:24.987 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.001+0000 7fbdaae5c640 1 -- 172.21.15.170:0/1338031296 >> 172.21.15.170:0/1338031296 conn(0x7fbda4066a00 msgr2=0x7fbda4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:24.987 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.001+0000 7fbdaae5c640 1 -- 172.21.15.170:0/1338031296 shutdown_connections 2024-03-29T11:14:24.987 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.001+0000 7fbdaae5c640 1 -- 172.21.15.170:0/1338031296 wait complete. 2024-03-29T11:14:24.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.002+0000 7fbdaae5c640 10 monclient: build_initial_monmap 2024-03-29T11:14:24.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.002+0000 7fbdaae5c640 10 monclient: monmap: 2024-03-29T11:14:24.992 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:14:24.992 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:24.992 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:14:24.992 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:14:24.992 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:24.992 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:24.992 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:14:24.992 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:14:24.992 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:14:24.992 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:24.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.003+0000 7fbdaae5c640 1 librados: starting msgr at 2024-03-29T11:14:24.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.003+0000 7fbdaae5c640 1 librados: starting objecter 2024-03-29T11:14:24.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.003+0000 7fbdaae5c640 1 Processor -- start 2024-03-29T11:14:24.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.003+0000 7fbdaae5c640 1 -- start start 2024-03-29T11:14:24.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.003+0000 7fbdaae5c640 1 librados: setting wanted keys 2024-03-29T11:14:24.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.003+0000 7fbdaae5c640 1 librados: calling monclient init 2024-03-29T11:14:24.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.003+0000 7fbdaae5c640 10 monclient: init 2024-03-29T11:14:24.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.004+0000 7fbdaae5c640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:24.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.004+0000 7fbdaae5c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:24.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.004+0000 7fbdaae5c640 10 monclient(hunting): picked mon.noname-c con 0x7fbda40a9d70 addr v1:172.21.15.170:6790/0 2024-03-29T11:14:24.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.004+0000 7fbdaae5c640 10 monclient(hunting): picked mon.noname-a con 0x7fbda413f650 addr v1:172.21.15.170:6789/0 2024-03-29T11:14:24.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.004+0000 7fbdaae5c640 10 monclient(hunting): picked mon.noname-b con 0x7fbda41457f0 addr v1:172.21.15.184:6789/0 2024-03-29T11:14:24.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.004+0000 7fbda3fff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fbda413f650 0x7fbda406f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:37778/0 (socket says 172.21.15.170:37778) 2024-03-29T11:14:24.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.004+0000 7fbda3fff640 1 -- 172.21.15.170:0/4171794857 learned_addr learned my addr 172.21.15.170:0/4171794857 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:14:24.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.004+0000 7fbdaae5c640 1 -- 172.21.15.170:0/4171794857 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda4070d50 con 0x7fbda413f650 2024-03-29T11:14:24.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.004+0000 7fbdaae5c640 1 -- 172.21.15.170:0/4171794857 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda4070fc0 con 0x7fbda41457f0 2024-03-29T11:14:24.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.004+0000 7fbdaae5c640 1 -- 172.21.15.170:0/4171794857 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda406f1f0 con 0x7fbda40a9d70 2024-03-29T11:14:24.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.004+0000 7fbdaae5c640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:24.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.004+0000 7fbdaae5c640 10 monclient(hunting): authenticate will time out at 1046.316694s 2024-03-29T11:14:24.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.004+0000 7fbda17fa640 10 client.?.objecter ms_handle_connect 0x7fbda40a9d70 2024-03-29T11:14:24.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.004+0000 7fbda17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:24.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283287985 0 0) 0x7fbda406f1f0 con 0x7fbda40a9d70 2024-03-29T11:14:24.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:14:24.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:24.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:24.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:14:24.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd88004eb0 con 0x7fbda40a9d70 2024-03-29T11:14:24.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 10 client.?.objecter ms_handle_connect 0x7fbda41457f0 2024-03-29T11:14:24.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:24.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935535217 0 0) 0x7fbda4070fc0 con 0x7fbda41457f0 2024-03-29T11:14:24.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:14:24.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:24.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:24.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:14:24.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd880051d0 con 0x7fbda41457f0 2024-03-29T11:14:24.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 10 client.?.objecter ms_handle_connect 0x7fbda413f650 2024-03-29T11:14:24.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:24.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3615773338 0 0) 0x7fbd88004eb0 con 0x7fbda40a9d70 2024-03-29T11:14:24.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd88005640 con 0x7fbda40a9d70 2024-03-29T11:14:24.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fbd98001b40 con 0x7fbda40a9d70 2024-03-29T11:14:24.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.005+0000 7fbda17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:24.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:24.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 10 monclient(hunting): dump: 2024-03-29T11:14:24.995 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:24.995 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:24.995 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:24.995 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:24.996 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:24.996 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:24.996 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:24.996 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:24.996 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:24.996 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:24.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:24.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 905552750 0 0) 0x7fbda4070d50 con 0x7fbda413f650 2024-03-29T11:14:24.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:14:24.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:24.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:24.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:14:24.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd88004eb0 con 0x7fbda413f650 2024-03-29T11:14:24.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1226347589 0 0) 0x7fbd88004eb0 con 0x7fbda413f650 2024-03-29T11:14:24.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbda4070d50 con 0x7fbda413f650 2024-03-29T11:14:24.997 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fbd90001930 con 0x7fbda413f650 2024-03-29T11:14:24.997 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:24.997 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:24.997 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 10 monclient(hunting): dump: 2024-03-29T11:14:24.997 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:24.997 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:24.997 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:24.997 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:24.997 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:24.997 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:24.997 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:24.997 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:24.997 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:24.997 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:24.997 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2149123784 0 0) 0x7fbd88005640 con 0x7fbda40a9d70 2024-03-29T11:14:24.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 >> v1:172.21.15.184:6789/0 conn(0x7fbda41457f0 legacy=0x7fbda4139470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:24.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 >> v1:172.21.15.170:6789/0 conn(0x7fbda413f650 legacy=0x7fbda406f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:24.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:14:24.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 1 monclient: found mon.c 2024-03-29T11:14:24.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:24.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:14:24.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda41413b0 con 0x7fbda40a9d70 2024-03-29T11:14:24.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 10 monclient: _finish_auth 0 2024-03-29T11:14:24.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:14:24.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbda17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:24.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbdaae5c640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:14:24.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbdaae5c640 10 monclient: _renew_subs 2024-03-29T11:14:24.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbdaae5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:14:24.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbdaae5c640 1 -- 172.21.15.170:0/4171794857 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda406ec40 con 0x7fbda40a9d70 2024-03-29T11:14:24.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbdaae5c640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:24.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbdaae5c640 10 monclient: _renew_subs 2024-03-29T11:14:24.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbdaae5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:14:24.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbdaae5c640 1 -- 172.21.15.170:0/4171794857 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda4139d60 con 0x7fbda40a9d70 2024-03-29T11:14:24.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.006+0000 7fbdaae5c640 1 librados: init done 2024-03-29T11:14:24.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.007+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd98004a90 con 0x7fbda40a9d70 2024-03-29T11:14:24.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.007+0000 7fbda17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:24.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.007+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fbd98001250 con 0x7fbda40a9d70 2024-03-29T11:14:24.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.007+0000 7fbda17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:24.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.007+0000 7fbda17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:24.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.007+0000 7fbda17fa640 10 monclient: dump: 2024-03-29T11:14:24.999 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:24.999 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:24.999 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:24.999 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:25.000 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:25.000 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:25.000 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:25.000 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:25.000 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:25.000 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:25.000 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.007+0000 7fbda2ffd640 4 set_mon_vals no callback set 2024-03-29T11:14:25.000 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.008+0000 7fbdaae5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:25.000 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.008+0000 7fbdaae5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:25.000 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.008+0000 7fbdaae5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:14:25.000 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.008+0000 7fbdaae5c640 1 -- 172.21.15.170:0/4171794857 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd84005080 con 0x7fbda40a9d70 2024-03-29T11:14:25.000 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.009+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fbd9801ac10 con 0x7fbda40a9d70 2024-03-29T11:14:25.000 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.010+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fbd980849e0 con 0x7fbda40a9d70 2024-03-29T11:14:25.000 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.010+0000 7fbda17fa640 10 client.4337.objecter ms_dispatch 0x7fbda4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:25.000 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.010+0000 7fbda17fa640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:25.001 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.010+0000 7fbda17fa640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:25.001 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.010+0000 7fbda17fa640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-29T11:14:25.001 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.010+0000 7fbda3fff640 10 monclient: get_auth_request con 0x7fbd8806e420 auth_method 0 2024-03-29T11:14:25.001 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.011+0000 7fbda17fa640 10 client.4337.objecter ms_handle_connect 0x7fbd8806e420 2024-03-29T11:14:25.001 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.018+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd980500f0 con 0x7fbda40a9d70 2024-03-29T11:14:25.001 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.018+0000 7fbda17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:25.001 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.018+0000 7fbda17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:25.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.397+0000 7fbdaae5c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:25.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.397+0000 7fbdaae5c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:25.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.397+0000 7fbdaae5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:14:25.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.397+0000 7fbdaae5c640 1 -- 172.21.15.170:0/4171794857 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd84005330 con 0x7fbda40a9d70 2024-03-29T11:14:25.376 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:14:25.376 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:25.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.398+0000 7fbda17fa640 1 -- 172.21.15.170:0/4171794857 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fbd980521f0 con 0x7fbda40a9d70 2024-03-29T11:14:25.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.398+0000 7fbda17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:25.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.398+0000 7fbda17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:25.380 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.402+0000 7fbdaae5c640 10 librados: watch_flush enter 2024-03-29T11:14:25.381 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.402+0000 7fbdaae5c640 10 librados: watch_flush exit 2024-03-29T11:14:25.381 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.402+0000 7fbdaae5c640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-29T11:14:25.381 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.402+0000 7fbdaae5c640 10 client.4337.objecter successfully canceled tick 2024-03-29T11:14:25.381 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.402+0000 7fbdaae5c640 1 -- 172.21.15.170:0/4171794857 >> v1:172.21.15.170:6816/3298994444 conn(0x7fbd8806e420 legacy=0x7fbd880708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:25.381 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.402+0000 7fbdaae5c640 10 monclient: shutdown 2024-03-29T11:14:25.381 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.402+0000 7fbdaae5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:25.381 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.402+0000 7fbdaae5c640 1 -- 172.21.15.170:0/4171794857 >> v1:172.21.15.170:6790/0 conn(0x7fbda40a9d70 legacy=0x7fbda4071a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:25.381 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.403+0000 7fbdaae5c640 1 -- 172.21.15.170:0/4171794857 shutdown_connections 2024-03-29T11:14:25.381 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.403+0000 7fbdaae5c640 1 -- 172.21.15.170:0/4171794857 >> 172.21.15.170:0/4171794857 conn(0x7fbda4066a00 msgr2=0x7fbda40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:25.381 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.403+0000 7fbdaae5c640 1 -- 172.21.15.170:0/4171794857 shutdown_connections 2024-03-29T11:14:25.381 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.403+0000 7fbdaae5c640 1 -- 172.21.15.170:0/4171794857 wait complete. 2024-03-29T11:14:25.381 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:25.403+0000 7fbdaae5c640 1 librados: shutdown 2024-03-29T11:14:25.399 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:32.205 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:32.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.360+0000 7f975136c640 10 monclient: get_monmap_and_config 2024-03-29T11:14:32.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.360+0000 7f975136c640 10 monclient: build_initial_monmap 2024-03-29T11:14:32.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.360+0000 7f975136c640 10 monclient: monmap: 2024-03-29T11:14:32.338 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:14:32.338 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:32.338 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:14:32.361889+0000 2024-03-29T11:14:32.339 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:14:32.361889+0000 2024-03-29T11:14:32.339 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:32.339 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:32.339 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:14:32.339 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:14:32.339 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:14:32.339 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:32.340 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.361+0000 7f975136c640 1 Processor -- start 2024-03-29T11:14:32.340 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.361+0000 7f975136c640 1 -- start start 2024-03-29T11:14:32.340 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.361+0000 7f975136c640 10 monclient: init 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.362+0000 7f975136c640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.362+0000 7f975136c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.362+0000 7f975136c640 10 monclient(hunting): picked mon.noname-a con 0x7f974c1457f0 addr v1:172.21.15.170:6789/0 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.362+0000 7f975136c640 10 monclient(hunting): picked mon.noname-c con 0x7f974c13f650 addr v1:172.21.15.184:6789/0 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.362+0000 7f975136c640 10 monclient(hunting): picked mon.noname-b con 0x7f974c0a9d70 addr v1:172.21.15.170:6790/0 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.362+0000 7f975136c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f974c0aaef0 con 0x7f974c1457f0 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.362+0000 7f975136c640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f974c145bd0 con 0x7f974c13f650 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.362+0000 7f975136c640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f974c145eb0 con 0x7f974c0a9d70 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.362+0000 7f975136c640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.362+0000 7f975136c640 10 monclient(hunting): authenticate will time out at 1053.674774s 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.363+0000 7f974b7fe640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f974c0a9d70 0x7f974c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:49682/0 (socket says 172.21.15.170:49682) 2024-03-29T11:14:32.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.363+0000 7f974b7fe640 1 -- 172.21.15.170:0/3166063169 learned_addr learned my addr 172.21.15.170:0/3166063169 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:14:32.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.363+0000 7f9749ffb640 1 -- 172.21.15.170:0/3166063169 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773397159 0 0) 0x7f974c145eb0 con 0x7f974c0a9d70 2024-03-29T11:14:32.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.363+0000 7f9749ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:32.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.363+0000 7f9749ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:32.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.363+0000 7f9749ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:32.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.363+0000 7f9749ffb640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:14:32.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.363+0000 7f9749ffb640 1 -- 172.21.15.170:0/3166063169 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9734003520 con 0x7f974c0a9d70 2024-03-29T11:14:32.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.364+0000 7f9749ffb640 1 -- 172.21.15.170:0/3166063169 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335539901 0 0) 0x7f974c145bd0 con 0x7f974c13f650 2024-03-29T11:14:32.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.364+0000 7f9749ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:32.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.364+0000 7f9749ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:32.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.364+0000 7f9749ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:32.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.364+0000 7f9749ffb640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:14:32.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.364+0000 7f9749ffb640 1 -- 172.21.15.170:0/3166063169 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f974c145eb0 con 0x7f974c13f650 2024-03-29T11:14:32.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.364+0000 7f9749ffb640 1 -- 172.21.15.170:0/3166063169 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 716117755 0 0) 0x7f9734003520 con 0x7f974c0a9d70 2024-03-29T11:14:32.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.364+0000 7f9749ffb640 1 -- 172.21.15.170:0/3166063169 >> v1:172.21.15.184:6789/0 conn(0x7f974c13f650 legacy=0x7f974c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:32.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.364+0000 7f9749ffb640 1 -- 172.21.15.170:0/3166063169 >> v1:172.21.15.170:6789/0 conn(0x7f974c1457f0 legacy=0x7f974c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:14:32.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f9749ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:14:32.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f9749ffb640 1 monclient: found mon.noname-b 2024-03-29T11:14:32.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f9749ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:32.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f9749ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:14:32.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f9749ffb640 1 -- 172.21.15.170:0/3166063169 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974c146340 con 0x7f974c0a9d70 2024-03-29T11:14:32.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f9749ffb640 10 monclient: _finish_auth 0 2024-03-29T11:14:32.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f9749ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:14:32.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f9749ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:32.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f9749ffb640 1 -- 172.21.15.170:0/3166063169 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9744001a10 con 0x7f974c0a9d70 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f975136c640 5 monclient: authenticate success, global_id 4340 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f975136c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f9749ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f9749ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f9749ffb640 10 monclient: dump: 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f9749ffb640 1 -- 172.21.15.170:0/3166063169 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9744001b70 con 0x7f974c0a9d70 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f975136c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f9749ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f9749ffb640 1 -- 172.21.15.170:0/3166063169 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9744001f80 con 0x7f974c0a9d70 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f975136c640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.365+0000 7f975136c640 4 set_mon_vals no callback set 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.366+0000 7f9749ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:32.355 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.366+0000 7f975136c640 10 monclient: shutdown 2024-03-29T11:14:32.356 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.366+0000 7f9749ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:32.356 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.366+0000 7f9749ffb640 10 monclient: dump: 2024-03-29T11:14:32.356 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:32.357 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:32.357 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:32.357 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:32.357 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:32.357 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:32.357 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:32.357 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:32.358 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:32.358 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:32.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.366+0000 7f975136c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:32.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.366+0000 7f975136c640 1 -- 172.21.15.170:0/3166063169 >> v1:172.21.15.170:6790/0 conn(0x7f974c0a9d70 legacy=0x7f974c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:32.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.367+0000 7f975136c640 1 -- 172.21.15.170:0/3166063169 shutdown_connections 2024-03-29T11:14:32.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.367+0000 7f975136c640 1 -- 172.21.15.170:0/3166063169 >> 172.21.15.170:0/3166063169 conn(0x7f974c066a00 msgr2=0x7f974c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:32.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.367+0000 7f975136c640 1 -- 172.21.15.170:0/3166063169 shutdown_connections 2024-03-29T11:14:32.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.367+0000 7f975136c640 1 -- 172.21.15.170:0/3166063169 wait complete. 2024-03-29T11:14:32.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.368+0000 7f975136c640 10 monclient: build_initial_monmap 2024-03-29T11:14:32.358 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.368+0000 7f975136c640 10 monclient: monmap: 2024-03-29T11:14:32.358 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:14:32.359 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:32.359 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:14:32.359 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:14:32.359 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:32.359 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:32.359 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:14:32.359 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:14:32.359 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:14:32.359 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:32.359 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.368+0000 7f975136c640 1 librados: starting msgr at 2024-03-29T11:14:32.359 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.368+0000 7f975136c640 1 librados: starting objecter 2024-03-29T11:14:32.359 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.369+0000 7f975136c640 1 Processor -- start 2024-03-29T11:14:32.359 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.369+0000 7f975136c640 1 -- start start 2024-03-29T11:14:32.360 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.369+0000 7f975136c640 1 librados: setting wanted keys 2024-03-29T11:14:32.360 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.369+0000 7f975136c640 1 librados: calling monclient init 2024-03-29T11:14:32.360 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.369+0000 7f975136c640 10 monclient: init 2024-03-29T11:14:32.360 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.370+0000 7f975136c640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:32.360 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.370+0000 7f975136c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:32.360 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.370+0000 7f975136c640 10 monclient(hunting): picked mon.noname-c con 0x7f974c0a9d70 addr v1:172.21.15.170:6790/0 2024-03-29T11:14:32.360 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.371+0000 7f974affd640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f974c0a9d70 0x7f974c1d6fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:51044/0 (socket says 172.21.15.170:51044) 2024-03-29T11:14:32.360 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.371+0000 7f974affd640 1 -- 172.21.15.170:0/743071682 learned_addr learned my addr 172.21.15.170:0/743071682 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:14:32.360 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.371+0000 7f975136c640 10 monclient(hunting): picked mon.noname-a con 0x7f974c13f650 addr v1:172.21.15.170:6789/0 2024-03-29T11:14:32.360 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.371+0000 7f97337fe640 10 client.?.objecter ms_handle_connect 0x7f974c0a9d70 2024-03-29T11:14:32.360 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f97337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:32.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f975136c640 10 monclient(hunting): picked mon.noname-b con 0x7f974c1457f0 addr v1:172.21.15.184:6789/0 2024-03-29T11:14:32.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f975136c640 1 -- 172.21.15.170:0/743071682 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f974c0ad910 con 0x7f974c13f650 2024-03-29T11:14:32.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f975136c640 1 -- 172.21.15.170:0/743071682 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f974c1d76d0 con 0x7f974c1457f0 2024-03-29T11:14:32.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f975136c640 1 -- 172.21.15.170:0/743071682 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f974c0b1800 con 0x7f974c0a9d70 2024-03-29T11:14:32.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f975136c640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:32.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f975136c640 10 monclient(hunting): authenticate will time out at 1053.684222s 2024-03-29T11:14:32.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f97337fe640 1 -- 172.21.15.170:0/743071682 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912650992 0 0) 0x7f974c0b1800 con 0x7f974c0a9d70 2024-03-29T11:14:32.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f97337fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:32.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f97337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:32.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f97337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:32.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f97337fe640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:14:32.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f97337fe640 1 -- 172.21.15.170:0/743071682 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9738004eb0 con 0x7f974c0a9d70 2024-03-29T11:14:32.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f97337fe640 1 -- 172.21.15.170:0/743071682 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3961370884 0 0) 0x7f9738004eb0 con 0x7f974c0a9d70 2024-03-29T11:14:32.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f97337fe640 1 -- 172.21.15.170:0/743071682 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f974c0b1800 con 0x7f974c0a9d70 2024-03-29T11:14:32.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f97337fe640 1 -- 172.21.15.170:0/743071682 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f973c0017d0 con 0x7f974c0a9d70 2024-03-29T11:14:32.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f97337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:32.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f97337fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:32.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f97337fe640 10 monclient(hunting): dump: 2024-03-29T11:14:32.363 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:32.363 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:32.363 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:32.363 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:32.363 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:32.363 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:32.363 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:32.363 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:32.363 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:32.364 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:32.364 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.372+0000 7f97337fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:32.364 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f97337fe640 10 client.?.objecter ms_handle_connect 0x7f974c1457f0 2024-03-29T11:14:32.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f97337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:32.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f97337fe640 1 -- 172.21.15.170:0/743071682 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3844030447 0 0) 0x7f974c0b1800 con 0x7f974c0a9d70 2024-03-29T11:14:32.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f97337fe640 1 -- 172.21.15.170:0/743071682 >> v1:172.21.15.184:6789/0 conn(0x7f974c1457f0 legacy=0x7f974c0b10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:32.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f97337fe640 1 -- 172.21.15.170:0/743071682 >> v1:172.21.15.170:6789/0 conn(0x7f974c13f650 legacy=0x7f974c0b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:32.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f97337fe640 10 monclient: _finish_hunting 0 2024-03-29T11:14:32.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f97337fe640 1 monclient: found mon.c 2024-03-29T11:14:32.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f97337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:32.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f97337fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:14:32.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f97337fe640 1 -- 172.21.15.170:0/743071682 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974c0ad5a0 con 0x7f974c0a9d70 2024-03-29T11:14:32.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f97337fe640 10 monclient: _finish_auth 0 2024-03-29T11:14:32.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f97337fe640 10 monclient: _check_auth_tickets 2024-03-29T11:14:32.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f97337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:32.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f975136c640 5 monclient: authenticate success, global_id 4343 2024-03-29T11:14:32.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f975136c640 10 monclient: _renew_subs 2024-03-29T11:14:32.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f975136c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:14:32.367 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f975136c640 1 -- 172.21.15.170:0/743071682 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974c0ad0e0 con 0x7f974c0a9d70 2024-03-29T11:14:32.367 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f975136c640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:32.367 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f975136c640 10 monclient: _renew_subs 2024-03-29T11:14:32.367 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f975136c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:14:32.367 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f975136c640 1 -- 172.21.15.170:0/743071682 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f974c142230 con 0x7f974c0a9d70 2024-03-29T11:14:32.367 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.373+0000 7f975136c640 1 librados: init done 2024-03-29T11:14:32.367 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.374+0000 7f975136c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:32.367 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.374+0000 7f975136c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:32.367 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.374+0000 7f975136c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:14:32.367 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.374+0000 7f975136c640 1 -- 172.21.15.170:0/743071682 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974c0af360 con 0x7f974c0a9d70 2024-03-29T11:14:32.368 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.374+0000 7f97337fe640 1 -- 172.21.15.170:0/743071682 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f973c001930 con 0x7f974c0a9d70 2024-03-29T11:14:32.368 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.374+0000 7f97337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:32.368 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.374+0000 7f97337fe640 1 -- 172.21.15.170:0/743071682 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f973c004a00 con 0x7f974c0a9d70 2024-03-29T11:14:32.368 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.374+0000 7f97337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:32.368 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.374+0000 7f97337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:32.368 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.374+0000 7f97337fe640 10 monclient: dump: 2024-03-29T11:14:32.368 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:32.368 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:32.368 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:32.368 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:32.368 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:32.369 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:32.369 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:32.369 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:32.369 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:32.369 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:32.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.375+0000 7f97497fa640 4 set_mon_vals no callback set 2024-03-29T11:14:32.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.375+0000 7f97337fe640 1 -- 172.21.15.170:0/743071682 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f973c01aa50 con 0x7f974c0a9d70 2024-03-29T11:14:32.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.376+0000 7f974a7fc640 10 monclient: get_auth_request con 0x7f973806ddd0 auth_method 0 2024-03-29T11:14:32.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.376+0000 7f97337fe640 1 -- 172.21.15.170:0/743071682 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f973c084830 con 0x7f974c0a9d70 2024-03-29T11:14:32.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.376+0000 7f97337fe640 10 client.4343.objecter ms_dispatch 0x7f974c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:32.370 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.376+0000 7f97337fe640 3 client.4343.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:32.370 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.376+0000 7f97337fe640 3 client.4343.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:32.370 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.376+0000 7f97337fe640 20 client.4343.objecter dump_active .. 0 homeless 2024-03-29T11:14:32.370 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.376+0000 7f97337fe640 10 client.4343.objecter ms_handle_connect 0x7f973806ddd0 2024-03-29T11:14:32.371 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.383+0000 7f97337fe640 1 -- 172.21.15.170:0/743071682 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f973c050170 con 0x7f974c0a9d70 2024-03-29T11:14:32.371 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.383+0000 7f97337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:32.371 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.383+0000 7f97337fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:32.674 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.696+0000 7f975136c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:32.674 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.696+0000 7f975136c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:32.674 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.696+0000 7f975136c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:14:32.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.696+0000 7f975136c640 1 -- 172.21.15.170:0/743071682 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f974c075f10 con 0x7f974c0a9d70 2024-03-29T11:14:32.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.697+0000 7f97337fe640 1 -- 172.21.15.170:0/743071682 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f973c050170 con 0x7f974c0a9d70 2024-03-29T11:14:32.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.697+0000 7f97337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:32.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.697+0000 7f97337fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:32.677 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:14:32.677 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:32.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.700+0000 7f975136c640 10 librados: watch_flush enter 2024-03-29T11:14:32.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.700+0000 7f975136c640 10 librados: watch_flush exit 2024-03-29T11:14:32.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.700+0000 7f975136c640 20 client.4343.objecter shutdown clearing up homeless session... 2024-03-29T11:14:32.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.700+0000 7f975136c640 10 client.4343.objecter successfully canceled tick 2024-03-29T11:14:32.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.700+0000 7f975136c640 1 -- 172.21.15.170:0/743071682 >> v1:172.21.15.170:6816/3298994444 conn(0x7f973806ddd0 legacy=0x7f9738070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:32.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.700+0000 7f975136c640 10 monclient: shutdown 2024-03-29T11:14:32.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.700+0000 7f975136c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:32.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.700+0000 7f975136c640 1 -- 172.21.15.170:0/743071682 >> v1:172.21.15.170:6790/0 conn(0x7f974c0a9d70 legacy=0x7f974c1d6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:32.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.700+0000 7f975136c640 1 -- 172.21.15.170:0/743071682 shutdown_connections 2024-03-29T11:14:32.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.700+0000 7f975136c640 1 -- 172.21.15.170:0/743071682 >> 172.21.15.170:0/743071682 conn(0x7f974c066a00 msgr2=0x7f974c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:32.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.700+0000 7f975136c640 1 -- 172.21.15.170:0/743071682 shutdown_connections 2024-03-29T11:14:32.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.700+0000 7f975136c640 1 -- 172.21.15.170:0/743071682 wait complete. 2024-03-29T11:14:32.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:32.700+0000 7f975136c640 1 librados: shutdown 2024-03-29T11:14:32.693 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:39.502 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:39.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.670+0000 7f374f59e640 10 monclient: get_monmap_and_config 2024-03-29T11:14:39.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.670+0000 7f374f59e640 10 monclient: build_initial_monmap 2024-03-29T11:14:39.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.670+0000 7f374f59e640 10 monclient: monmap: 2024-03-29T11:14:39.649 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:14:39.649 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:39.649 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:14:39.671772+0000 2024-03-29T11:14:39.649 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:14:39.671772+0000 2024-03-29T11:14:39.650 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:39.650 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:39.650 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:14:39.650 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:14:39.650 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:14:39.650 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:39.650 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.671+0000 7f374f59e640 1 Processor -- start 2024-03-29T11:14:39.650 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.671+0000 7f374f59e640 1 -- start start 2024-03-29T11:14:39.650 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.671+0000 7f374f59e640 10 monclient: init 2024-03-29T11:14:39.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.672+0000 7f374f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:39.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.672+0000 7f374f59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:14:39.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.672+0000 7f374f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f37500a9fe0 addr v1:172.21.15.184:6789/0 2024-03-29T11:14:39.652 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.672+0000 7f374f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f37500af600 addr v1:172.21.15.170:6789/0 2024-03-29T11:14:39.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.674+0000 7f374f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f37500ad340 addr v1:172.21.15.170:6790/0 2024-03-29T11:14:39.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.674+0000 7f374f59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37500aaec0 con 0x7f37500af600 2024-03-29T11:14:39.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.674+0000 7f374e59c640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f37500a9fe0 0x7f375013b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:57768/0 (socket says 172.21.15.170:57768) 2024-03-29T11:14:39.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.674+0000 7f374f59e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37500ab6a0 con 0x7f37500a9fe0 2024-03-29T11:14:39.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.674+0000 7f374e59c640 1 -- 172.21.15.170:0/2400641390 learned_addr learned my addr 172.21.15.170:0/2400641390 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:14:39.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.674+0000 7f374f59e640 1 -- 172.21.15.170:0/2400641390 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37500b00a0 con 0x7f37500ad340 2024-03-29T11:14:39.654 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.674+0000 7f374f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:39.654 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.674+0000 7f374f59e640 10 monclient(hunting): authenticate will time out at 1060.986470s 2024-03-29T11:14:39.654 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.674+0000 7f374d59a640 1 -- 172.21.15.170:0/2400641390 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85055307 0 0) 0x7f37500ab6a0 con 0x7f37500a9fe0 2024-03-29T11:14:39.654 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.675+0000 7f374d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:14:39.654 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.675+0000 7f374d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:39.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.675+0000 7f374d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:39.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.675+0000 7f374d59a640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:14:39.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.675+0000 7f374d59a640 1 -- 172.21.15.170:0/2400641390 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f373c003520 con 0x7f37500a9fe0 2024-03-29T11:14:39.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.675+0000 7f374d59a640 1 -- 172.21.15.170:0/2400641390 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2351318079 0 0) 0x7f373c003520 con 0x7f37500a9fe0 2024-03-29T11:14:39.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.675+0000 7f374d59a640 1 -- 172.21.15.170:0/2400641390 >> v1:172.21.15.170:6790/0 conn(0x7f37500ad340 legacy=0x7f3750149c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:14:39.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.676+0000 7f374d59a640 1 -- 172.21.15.170:0/2400641390 >> v1:172.21.15.170:6789/0 conn(0x7f37500af600 legacy=0x7f37500acc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:14:39.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.677+0000 7f374d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:14:39.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.677+0000 7f374d59a640 1 monclient: found mon.noname-c 2024-03-29T11:14:39.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.677+0000 7f374d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:39.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.677+0000 7f374d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:14:39.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.677+0000 7f374d59a640 1 -- 172.21.15.170:0/2400641390 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375013beb0 con 0x7f37500a9fe0 2024-03-29T11:14:39.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.677+0000 7f374d59a640 10 monclient: _finish_auth 0 2024-03-29T11:14:39.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.677+0000 7f374d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:14:39.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.677+0000 7f374d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:39.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.677+0000 7f374d59a640 1 -- 172.21.15.170:0/2400641390 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3748001690 con 0x7f37500a9fe0 2024-03-29T11:14:39.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.677+0000 7f374f59e640 5 monclient: authenticate success, global_id 4321 2024-03-29T11:14:39.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.677+0000 7f374f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:39.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.677+0000 7f374d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:39.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.677+0000 7f374d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:39.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.677+0000 7f374d59a640 10 monclient: dump: 2024-03-29T11:14:39.659 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:39.659 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:39.660 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:39.660 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:39.660 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:39.660 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:39.660 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:39.660 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:39.660 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:39.660 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:39.660 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.677+0000 7f374f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:39.660 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.678+0000 7f374d59a640 1 -- 172.21.15.170:0/2400641390 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3748001920 con 0x7f37500a9fe0 2024-03-29T11:14:39.660 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.678+0000 7f374d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:39.661 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.678+0000 7f374d59a640 1 -- 172.21.15.170:0/2400641390 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3748001cb0 con 0x7f37500a9fe0 2024-03-29T11:14:39.661 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.678+0000 7f374f59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:39.661 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.678+0000 7f374f59e640 4 set_mon_vals no callback set 2024-03-29T11:14:39.661 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.679+0000 7f374d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:39.661 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.679+0000 7f374d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:39.661 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.679+0000 7f374d59a640 10 monclient: dump: 2024-03-29T11:14:39.661 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:39.661 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:39.662 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:39.662 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:39.662 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:39.662 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:39.662 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:39.662 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:39.662 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:39.662 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:39.662 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.679+0000 7f374f59e640 10 monclient: shutdown 2024-03-29T11:14:39.663 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.679+0000 7f374f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:39.663 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.679+0000 7f374f59e640 1 -- 172.21.15.170:0/2400641390 >> v1:172.21.15.184:6789/0 conn(0x7f37500a9fe0 legacy=0x7f375013b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:39.665 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.683+0000 7f374f59e640 1 -- 172.21.15.170:0/2400641390 shutdown_connections 2024-03-29T11:14:39.665 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.683+0000 7f374f59e640 1 -- 172.21.15.170:0/2400641390 >> 172.21.15.170:0/2400641390 conn(0x7f37500673e0 msgr2=0x7f37500b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:39.665 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.683+0000 7f374f59e640 1 -- 172.21.15.170:0/2400641390 shutdown_connections 2024-03-29T11:14:39.665 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.683+0000 7f374f59e640 1 -- 172.21.15.170:0/2400641390 wait complete. 2024-03-29T11:14:39.665 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.684+0000 7f374f59e640 10 monclient: build_initial_monmap 2024-03-29T11:14:39.665 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.684+0000 7f374f59e640 10 monclient: monmap: 2024-03-29T11:14:39.665 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:14:39.665 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:39.665 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:14:39.665 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:14:39.665 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:39.665 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:39.665 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:14:39.665 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:14:39.666 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:14:39.666 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:39.666 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.685+0000 7f374f59e640 1 librados: starting msgr at 2024-03-29T11:14:39.666 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.685+0000 7f374f59e640 1 librados: starting objecter 2024-03-29T11:14:39.666 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.685+0000 7f374f59e640 1 Processor -- start 2024-03-29T11:14:39.666 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.685+0000 7f374f59e640 1 -- start start 2024-03-29T11:14:39.666 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.685+0000 7f374f59e640 1 librados: setting wanted keys 2024-03-29T11:14:39.666 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.685+0000 7f374f59e640 1 librados: calling monclient init 2024-03-29T11:14:39.666 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.685+0000 7f374f59e640 10 monclient: init 2024-03-29T11:14:39.666 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.686+0000 7f374f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:39.666 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.686+0000 7f374f59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:14:39.666 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.686+0000 7f374f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f37500a9fe0 addr v1:172.21.15.170:6790/0 2024-03-29T11:14:39.666 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.686+0000 7f374f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f37500ad340 addr v1:172.21.15.184:6789/0 2024-03-29T11:14:39.667 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.687+0000 7f374dd9b640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f37500ad340 0x7f3750136960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:58360/0 (socket says 172.21.15.170:58360) 2024-03-29T11:14:39.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.690+0000 7f374f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f37500af600 addr v1:172.21.15.170:6789/0 2024-03-29T11:14:39.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.690+0000 7f374f59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375014d830 con 0x7f37500af600 2024-03-29T11:14:39.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.690+0000 7f374f59e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375006ddd0 con 0x7f37500ad340 2024-03-29T11:14:39.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.691+0000 7f374f59e640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375006e0b0 con 0x7f37500a9fe0 2024-03-29T11:14:39.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.691+0000 7f374f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:39.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.691+0000 7f374f59e640 10 monclient(hunting): authenticate will time out at 1061.003167s 2024-03-29T11:14:39.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.691+0000 7f374dd9b640 1 -- 172.21.15.170:0/1404505579 learned_addr learned my addr 172.21.15.170:0/1404505579 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:14:39.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.691+0000 7f374e59c640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f37500a9fe0 0x7f375014dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:46453/0 (socket says 172.21.15.170:46453) 2024-03-29T11:14:39.671 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.693+0000 7f3746ffd640 10 client.?.objecter ms_handle_connect 0x7f37500a9fe0 2024-03-29T11:14:39.671 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.693+0000 7f3746ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:39.671 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.693+0000 7f3746ffd640 10 client.?.objecter ms_handle_connect 0x7f37500ad340 2024-03-29T11:14:39.671 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.693+0000 7f3746ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.693+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4207591819 0 0) 0x7f375006e0b0 con 0x7f37500a9fe0 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.693+0000 7f3746ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.693+0000 7f3746ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.693+0000 7f3746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.693+0000 7f3746ffd640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.693+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3730004eb0 con 0x7f37500a9fe0 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.693+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469278432 0 0) 0x7f375006ddd0 con 0x7f37500ad340 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.693+0000 7f3746ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.693+0000 7f3746ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.693+0000 7f3746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.693+0000 7f3746ffd640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.693+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f375006e0b0 con 0x7f37500ad340 2024-03-29T11:14:39.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 445816653 0 0) 0x7f3730004eb0 con 0x7f37500a9fe0 2024-03-29T11:14:39.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f375006ddd0 con 0x7f37500a9fe0 2024-03-29T11:14:39.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3748001550 con 0x7f37500a9fe0 2024-03-29T11:14:39.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:39.674 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:39.674 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:39.674 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:39.674 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 10 client.?.objecter ms_handle_connect 0x7f37500af600 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1492841133 0 0) 0x7f375006e0b0 con 0x7f37500ad340 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3730004eb0 con 0x7f37500ad340 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3738001710 con 0x7f37500ad340 2024-03-29T11:14:39.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:39.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:39.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:39.676 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:39.676 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:39.676 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:39.676 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:39.676 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:39.676 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:39.676 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:39.676 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:39.676 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:39.676 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:39.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 770207744 0 0) 0x7f375006ddd0 con 0x7f37500a9fe0 2024-03-29T11:14:39.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 >> v1:172.21.15.184:6789/0 conn(0x7f37500ad340 legacy=0x7f3750136960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:39.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 >> v1:172.21.15.170:6789/0 conn(0x7f37500af600 legacy=0x7f37501dfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:39.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:14:39.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 1 monclient: found mon.c 2024-03-29T11:14:39.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:39.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:14:39.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3750139230 con 0x7f37500a9fe0 2024-03-29T11:14:39.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 10 monclient: _finish_auth 0 2024-03-29T11:14:39.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:14:39.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:39.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f374f59e640 5 monclient: authenticate success, global_id 4346 2024-03-29T11:14:39.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f374f59e640 10 monclient: _renew_subs 2024-03-29T11:14:39.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f374f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:14:39.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f374f59e640 1 -- 172.21.15.170:0/1404505579 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f375014e3d0 con 0x7f37500a9fe0 2024-03-29T11:14:39.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3748001770 con 0x7f37500a9fe0 2024-03-29T11:14:39.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3748004d90 con 0x7f37500a9fe0 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f3746ffd640 10 monclient: dump: 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:39.679 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.694+0000 7f374f59e640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:39.679 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.695+0000 7f374f59e640 10 monclient: _renew_subs 2024-03-29T11:14:39.679 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.695+0000 7f374f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:14:39.679 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.695+0000 7f374f59e640 1 -- 172.21.15.170:0/1404505579 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f375014e720 con 0x7f37500a9fe0 2024-03-29T11:14:39.679 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.695+0000 7f374f59e640 1 librados: init done 2024-03-29T11:14:39.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.697+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f374801ad60 con 0x7f37500a9fe0 2024-03-29T11:14:39.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.697+0000 7f374dd9b640 10 monclient: get_auth_request con 0x7f373006de90 auth_method 0 2024-03-29T11:14:39.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.697+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f3748084ab0 con 0x7f37500a9fe0 2024-03-29T11:14:39.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.697+0000 7f3746ffd640 10 client.4346.objecter ms_dispatch 0x7f3750000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:39.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.697+0000 7f3746ffd640 3 client.4346.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:39.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.697+0000 7f3746ffd640 3 client.4346.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:39.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.697+0000 7f3746ffd640 20 client.4346.objecter dump_active .. 0 homeless 2024-03-29T11:14:39.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.697+0000 7f374f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:39.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.697+0000 7f374f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:39.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.697+0000 7f374f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:14:39.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.697+0000 7f374f59e640 1 -- 172.21.15.170:0/1404505579 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f375004f730 con 0x7f37500a9fe0 2024-03-29T11:14:39.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.698+0000 7f3746ffd640 10 client.4346.objecter ms_handle_connect 0x7f373006de90 2024-03-29T11:14:39.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.698+0000 7f3747fff640 4 set_mon_vals no callback set 2024-03-29T11:14:39.686 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.706+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3748050390 con 0x7f37500a9fe0 2024-03-29T11:14:39.686 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.706+0000 7f3746ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:39.686 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:39.706+0000 7f3746ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:40.006 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.027+0000 7f374f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:40.006 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.027+0000 7f374f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:40.006 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.027+0000 7f374f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:14:40.006 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.027+0000 7f374f59e640 1 -- 172.21.15.170:0/1404505579 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37500b03d0 con 0x7f37500a9fe0 2024-03-29T11:14:40.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.028+0000 7f3746ffd640 1 -- 172.21.15.170:0/1404505579 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f37480522c0 con 0x7f37500a9fe0 2024-03-29T11:14:40.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.028+0000 7f3746ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:40.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.028+0000 7f3746ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:40.007 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:14:40.007 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:40.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.031+0000 7f374f59e640 10 librados: watch_flush enter 2024-03-29T11:14:40.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.032+0000 7f374f59e640 10 librados: watch_flush exit 2024-03-29T11:14:40.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.032+0000 7f374f59e640 20 client.4346.objecter shutdown clearing up homeless session... 2024-03-29T11:14:40.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.032+0000 7f374f59e640 10 client.4346.objecter successfully canceled tick 2024-03-29T11:14:40.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.032+0000 7f374f59e640 1 -- 172.21.15.170:0/1404505579 >> v1:172.21.15.170:6816/3298994444 conn(0x7f373006de90 legacy=0x7f3730070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:40.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.032+0000 7f374f59e640 10 monclient: shutdown 2024-03-29T11:14:40.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.032+0000 7f374f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:40.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.032+0000 7f374f59e640 1 -- 172.21.15.170:0/1404505579 >> v1:172.21.15.170:6790/0 conn(0x7f37500a9fe0 legacy=0x7f375014dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:40.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.032+0000 7f374f59e640 1 -- 172.21.15.170:0/1404505579 shutdown_connections 2024-03-29T11:14:40.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.032+0000 7f374f59e640 1 -- 172.21.15.170:0/1404505579 >> 172.21.15.170:0/1404505579 conn(0x7f37500673e0 msgr2=0x7f3750069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:40.013 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.034+0000 7f374f59e640 1 -- 172.21.15.170:0/1404505579 shutdown_connections 2024-03-29T11:14:40.013 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.034+0000 7f374f59e640 1 -- 172.21.15.170:0/1404505579 wait complete. 2024-03-29T11:14:40.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:40.035+0000 7f374f59e640 1 librados: shutdown 2024-03-29T11:14:40.032 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:46.841 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:46.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.016+0000 7ff5a17b6640 10 monclient: get_monmap_and_config 2024-03-29T11:14:46.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.016+0000 7ff5a17b6640 10 monclient: build_initial_monmap 2024-03-29T11:14:46.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.016+0000 7ff5a17b6640 10 monclient: monmap: 2024-03-29T11:14:46.994 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:14:46.994 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:46.994 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:14:47.017114+0000 2024-03-29T11:14:46.994 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:14:47.017114+0000 2024-03-29T11:14:46.994 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:46.994 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:46.994 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:14:46.994 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:14:46.995 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:14:46.995 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:46.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.018+0000 7ff5a17b6640 1 Processor -- start 2024-03-29T11:14:46.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.018+0000 7ff5a17b6640 1 -- start start 2024-03-29T11:14:46.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.018+0000 7ff5a17b6640 10 monclient: init 2024-03-29T11:14:46.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.019+0000 7ff5a17b6640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:46.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.019+0000 7ff5a17b6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:46.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.019+0000 7ff5a17b6640 10 monclient(hunting): picked mon.noname-b con 0x7ff59c0a9ac0 addr v1:172.21.15.170:6790/0 2024-03-29T11:14:46.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.020+0000 7ff5a17b6640 10 monclient(hunting): picked mon.noname-a con 0x7ff59c13ee90 addr v1:172.21.15.170:6789/0 2024-03-29T11:14:46.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.020+0000 7ff5a17b6640 10 monclient(hunting): picked mon.noname-c con 0x7ff59c0af900 addr v1:172.21.15.184:6789/0 2024-03-29T11:14:46.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.020+0000 7ff5a17b6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c0afd10 con 0x7ff59c13ee90 2024-03-29T11:14:46.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.020+0000 7ff5a17b6640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c0af6f0 con 0x7ff59c0af900 2024-03-29T11:14:46.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.020+0000 7ff5a17b6640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c0aac80 con 0x7ff59c0a9ac0 2024-03-29T11:14:46.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.020+0000 7ff5a17b6640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:46.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.020+0000 7ff5a17b6640 10 monclient(hunting): authenticate will time out at 1068.331550s 2024-03-29T11:14:47.003 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.026+0000 7ff59b7fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7ff59c0af900 0x7ff59c149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:35967/0 (socket says 172.21.15.170:35967) 2024-03-29T11:14:47.003 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.026+0000 7ff59b7fe640 1 -- 172.21.15.170:0/451787849 learned_addr learned my addr 172.21.15.170:0/451787849 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:14:47.003 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.026+0000 7ff599ffb640 1 -- 172.21.15.170:0/451787849 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473329789 0 0) 0x7ff59c0af6f0 con 0x7ff59c0af900 2024-03-29T11:14:47.003 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.026+0000 7ff599ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:47.004 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.026+0000 7ff599ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:47.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.030+0000 7ff599ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:47.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.030+0000 7ff599ffb640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:14:47.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.030+0000 7ff599ffb640 1 -- 172.21.15.170:0/451787849 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff584003520 con 0x7ff59c0af900 2024-03-29T11:14:47.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.031+0000 7ff599ffb640 1 -- 172.21.15.170:0/451787849 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054889305 0 0) 0x7ff59c0afd10 con 0x7ff59c13ee90 2024-03-29T11:14:47.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.031+0000 7ff599ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:47.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.031+0000 7ff599ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:47.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.031+0000 7ff599ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:47.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.031+0000 7ff599ffb640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:14:47.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.031+0000 7ff599ffb640 1 -- 172.21.15.170:0/451787849 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff59c0af6f0 con 0x7ff59c13ee90 2024-03-29T11:14:47.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.031+0000 7ff599ffb640 1 -- 172.21.15.170:0/451787849 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193974654 0 0) 0x7ff59c0aac80 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.031+0000 7ff599ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:47.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.031+0000 7ff599ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:47.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.031+0000 7ff599ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:47.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.031+0000 7ff599ffb640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:14:47.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.031+0000 7ff599ffb640 1 -- 172.21.15.170:0/451787849 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff59c0afd10 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.032+0000 7ff599ffb640 1 -- 172.21.15.170:0/451787849 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2974957032 0 0) 0x7ff584003520 con 0x7ff59c0af900 2024-03-29T11:14:47.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.032+0000 7ff599ffb640 1 -- 172.21.15.170:0/451787849 >> v1:172.21.15.170:6790/0 conn(0x7ff59c0a9ac0 legacy=0x7ff59c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:47.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.032+0000 7ff599ffb640 1 -- 172.21.15.170:0/451787849 >> v1:172.21.15.170:6789/0 conn(0x7ff59c13ee90 legacy=0x7ff59c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:47.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.032+0000 7ff599ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:14:47.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.032+0000 7ff599ffb640 1 monclient: found mon.noname-c 2024-03-29T11:14:47.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.032+0000 7ff599ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:47.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.032+0000 7ff599ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:14:47.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.032+0000 7ff599ffb640 1 -- 172.21.15.170:0/451787849 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c0aaf60 con 0x7ff59c0af900 2024-03-29T11:14:47.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.032+0000 7ff599ffb640 10 monclient: _finish_auth 0 2024-03-29T11:14:47.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.032+0000 7ff599ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:14:47.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.032+0000 7ff599ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:47.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.032+0000 7ff599ffb640 1 -- 172.21.15.170:0/451787849 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7ff594001a10 con 0x7ff59c0af900 2024-03-29T11:14:47.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.032+0000 7ff599ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:47.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.032+0000 7ff599ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:47.012 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.032+0000 7ff599ffb640 10 monclient: dump: 2024-03-29T11:14:47.012 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:47.012 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:47.012 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:47.012 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:47.012 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:47.012 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:47.012 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:47.012 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:47.012 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:47.012 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:47.012 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.032+0000 7ff5a17b6640 5 monclient: authenticate success, global_id 4327 2024-03-29T11:14:47.012 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.032+0000 7ff5a17b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:47.013 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.033+0000 7ff599ffb640 1 -- 172.21.15.170:0/451787849 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff594001b70 con 0x7ff59c0af900 2024-03-29T11:14:47.013 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.033+0000 7ff599ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:47.013 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.033+0000 7ff599ffb640 1 -- 172.21.15.170:0/451787849 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7ff594001fc0 con 0x7ff59c0af900 2024-03-29T11:14:47.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.033+0000 7ff5a17b6640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:47.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.034+0000 7ff5a17b6640 4 set_mon_vals no callback set 2024-03-29T11:14:47.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.034+0000 7ff599ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:47.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.034+0000 7ff5a17b6640 10 monclient: shutdown 2024-03-29T11:14:47.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.034+0000 7ff599ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:47.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.034+0000 7ff599ffb640 10 monclient: dump: 2024-03-29T11:14:47.014 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:47.014 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:47.014 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:47.014 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:47.014 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:47.014 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:47.014 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:47.014 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:47.015 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:47.015 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:47.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.035+0000 7ff5a17b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:47.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.035+0000 7ff5a17b6640 1 -- 172.21.15.170:0/451787849 >> v1:172.21.15.184:6789/0 conn(0x7ff59c0af900 legacy=0x7ff59c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:47.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.035+0000 7ff5a17b6640 1 -- 172.21.15.170:0/451787849 shutdown_connections 2024-03-29T11:14:47.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.035+0000 7ff5a17b6640 1 -- 172.21.15.170:0/451787849 >> 172.21.15.170:0/451787849 conn(0x7ff59c0b35e0 msgr2=0x7ff59c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:47.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.036+0000 7ff5a17b6640 1 -- 172.21.15.170:0/451787849 shutdown_connections 2024-03-29T11:14:47.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.038+0000 7ff5a17b6640 1 -- 172.21.15.170:0/451787849 wait complete. 2024-03-29T11:14:47.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.039+0000 7ff5a17b6640 10 monclient: build_initial_monmap 2024-03-29T11:14:47.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.039+0000 7ff5a17b6640 10 monclient: monmap: 2024-03-29T11:14:47.016 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:14:47.016 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:47.016 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:14:47.016 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:14:47.016 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:47.017 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:47.017 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:14:47.017 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:14:47.017 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:14:47.017 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:47.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.040+0000 7ff5a17b6640 1 librados: starting msgr at 2024-03-29T11:14:47.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.040+0000 7ff5a17b6640 1 librados: starting objecter 2024-03-29T11:14:47.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.040+0000 7ff5a17b6640 1 Processor -- start 2024-03-29T11:14:47.018 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.041+0000 7ff5a17b6640 1 -- start start 2024-03-29T11:14:47.018 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.041+0000 7ff5a17b6640 1 librados: setting wanted keys 2024-03-29T11:14:47.018 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.041+0000 7ff5a17b6640 1 librados: calling monclient init 2024-03-29T11:14:47.018 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.041+0000 7ff5a17b6640 10 monclient: init 2024-03-29T11:14:47.019 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.042+0000 7ff5a17b6640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:47.019 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.042+0000 7ff5a17b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:14:47.019 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.042+0000 7ff5a17b6640 10 monclient(hunting): picked mon.noname-b con 0x7ff59c0a9ac0 addr v1:172.21.15.184:6789/0 2024-03-29T11:14:47.019 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.042+0000 7ff5a17b6640 10 monclient(hunting): picked mon.noname-c con 0x7ff59c0af900 addr v1:172.21.15.170:6790/0 2024-03-29T11:14:47.019 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.042+0000 7ff5a17b6640 10 monclient(hunting): picked mon.noname-a con 0x7ff59c13ee90 addr v1:172.21.15.170:6789/0 2024-03-29T11:14:47.019 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.042+0000 7ff5a17b6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c13e250 con 0x7ff59c13ee90 2024-03-29T11:14:47.019 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.042+0000 7ff5a17b6640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c13dd90 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.042+0000 7ff5a17b6640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c13df20 con 0x7ff59c0af900 2024-03-29T11:14:47.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.042+0000 7ff5a17b6640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:47.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.042+0000 7ff5a17b6640 10 monclient(hunting): authenticate will time out at 1068.353507s 2024-03-29T11:14:47.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.047+0000 7ff59a7fc640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7ff59c0af900 0x7ff59c13c300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:56091/0 (socket says 172.21.15.170:56091) 2024-03-29T11:14:47.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.047+0000 7ff59a7fc640 1 -- 172.21.15.170:0/825872472 learned_addr learned my addr 172.21.15.170:0/825872472 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:14:47.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.048+0000 7ff5837fe640 10 client.?.objecter ms_handle_connect 0x7ff59c0af900 2024-03-29T11:14:47.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.048+0000 7ff5837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:47.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.048+0000 7ff5837fe640 1 -- 172.21.15.170:0/825872472 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512325525 0 0) 0x7ff59c13df20 con 0x7ff59c0af900 2024-03-29T11:14:47.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.048+0000 7ff5837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.048+0000 7ff5837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.048+0000 7ff5837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.048+0000 7ff5837fe640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.048+0000 7ff5837fe640 1 -- 172.21.15.170:0/825872472 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff588004eb0 con 0x7ff59c0af900 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.049+0000 7ff5837fe640 10 client.?.objecter ms_handle_connect 0x7ff59c0a9ac0 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.049+0000 7ff5837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.049+0000 7ff5837fe640 1 -- 172.21.15.170:0/825872472 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171530769 0 0) 0x7ff59c13dd90 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.049+0000 7ff5837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.049+0000 7ff5837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.049+0000 7ff5837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.049+0000 7ff5837fe640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.049+0000 7ff5837fe640 1 -- 172.21.15.170:0/825872472 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff59c13df20 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.049+0000 7ff5837fe640 1 -- 172.21.15.170:0/825872472 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4179763613 0 0) 0x7ff59c13df20 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.050+0000 7ff5837fe640 1 -- 172.21.15.170:0/825872472 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff59c13dd90 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.050+0000 7ff5837fe640 1 -- 172.21.15.170:0/825872472 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7ff5900017e0 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.050+0000 7ff5837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:47.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.050+0000 7ff5837fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:47.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.050+0000 7ff5837fe640 10 monclient(hunting): dump: 2024-03-29T11:14:47.028 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:47.028 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:47.028 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:47.028 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:47.028 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:47.028 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:47.028 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:47.028 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:47.028 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:47.028 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:47.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.050+0000 7ff5837fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:47.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.050+0000 7ff5837fe640 1 -- 172.21.15.170:0/825872472 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1191773670 0 0) 0x7ff59c13dd90 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.050+0000 7ff5837fe640 1 -- 172.21.15.170:0/825872472 >> v1:172.21.15.170:6790/0 conn(0x7ff59c0af900 legacy=0x7ff59c13c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:47.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.050+0000 7ff5837fe640 1 -- 172.21.15.170:0/825872472 >> v1:172.21.15.170:6789/0 conn(0x7ff59c13ee90 legacy=0x7ff59c1396d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:14:47.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.050+0000 7ff5837fe640 10 monclient: _finish_hunting 0 2024-03-29T11:14:47.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.050+0000 7ff5837fe640 1 monclient: found mon.b 2024-03-29T11:14:47.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.051+0000 7ff5837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.051+0000 7ff5837fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.051+0000 7ff5837fe640 1 -- 172.21.15.170:0/825872472 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c0b2f70 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.051+0000 7ff5837fe640 10 monclient: _finish_auth 0 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.051+0000 7ff5837fe640 10 monclient: _check_auth_tickets 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.051+0000 7ff5837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.051+0000 7ff5a17b6640 5 monclient: authenticate success, global_id 4330 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.051+0000 7ff5a17b6640 10 monclient: _renew_subs 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.051+0000 7ff5a17b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.051+0000 7ff5a17b6640 1 -- 172.21.15.170:0/825872472 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff59c0b3220 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.051+0000 7ff5a17b6640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.051+0000 7ff5a17b6640 10 monclient: _renew_subs 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.051+0000 7ff5a17b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.051+0000 7ff5a17b6640 1 -- 172.21.15.170:0/825872472 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff59c132770 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.051+0000 7ff5a17b6640 1 librados: init done 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.053+0000 7ff5837fe640 1 -- 172.21.15.170:0/825872472 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff590001cf0 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.053+0000 7ff5837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.053+0000 7ff598ff9640 4 set_mon_vals no callback set 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.054+0000 7ff5837fe640 1 -- 172.21.15.170:0/825872472 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7ff5900037b0 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.054+0000 7ff5837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.054+0000 7ff5837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.054+0000 7ff5837fe640 10 monclient: dump: 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:47.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.055+0000 7ff5837fe640 1 -- 172.21.15.170:0/825872472 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7ff590003950 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.055+0000 7ff5817fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:47.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.055+0000 7ff5817fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.055+0000 7ff5817fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.055+0000 7ff5817fa640 1 -- 172.21.15.170:0/825872472 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff57c005080 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.056+0000 7ff59a7fc640 10 monclient: get_auth_request con 0x7ff58806dcb0 auth_method 0 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.056+0000 7ff5837fe640 1 -- 172.21.15.170:0/825872472 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7ff590083780 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.056+0000 7ff5837fe640 10 client.4330.objecter ms_dispatch 0x7ff59c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.056+0000 7ff5837fe640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.056+0000 7ff5837fe640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.056+0000 7ff5837fe640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.056+0000 7ff5837fe640 10 client.4330.objecter ms_handle_connect 0x7ff58806dcb0 2024-03-29T11:14:47.047 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.066+0000 7ff5837fe640 1 -- 172.21.15.170:0/825872472 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff59004f060 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.048 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.066+0000 7ff5837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:47.048 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.066+0000 7ff5837fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:47.371 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.393+0000 7ff5817fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:47.371 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.393+0000 7ff5817fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:47.371 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.393+0000 7ff5817fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:47.371 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.393+0000 7ff5817fa640 1 -- 172.21.15.170:0/825872472 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff57c005330 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.372 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.395+0000 7ff5837fe640 1 -- 172.21.15.170:0/825872472 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7ff590050f90 con 0x7ff59c0a9ac0 2024-03-29T11:14:47.372 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.395+0000 7ff5837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:47.372 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.395+0000 7ff5837fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:47.374 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:14:47.374 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:47.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.401+0000 7ff5817fa640 10 librados: watch_flush enter 2024-03-29T11:14:47.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.402+0000 7ff5817fa640 10 librados: watch_flush exit 2024-03-29T11:14:47.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.402+0000 7ff5817fa640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-29T11:14:47.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.402+0000 7ff5817fa640 10 client.4330.objecter successfully canceled tick 2024-03-29T11:14:47.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.402+0000 7ff5817fa640 1 -- 172.21.15.170:0/825872472 >> v1:172.21.15.170:6816/3298994444 conn(0x7ff58806dcb0 legacy=0x7ff588070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:47.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.402+0000 7ff5817fa640 10 monclient: shutdown 2024-03-29T11:14:47.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.402+0000 7ff5817fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:47.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.402+0000 7ff5817fa640 1 -- 172.21.15.170:0/825872472 >> v1:172.21.15.184:6789/0 conn(0x7ff59c0a9ac0 legacy=0x7ff59c1da790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:47.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.404+0000 7ff5817fa640 1 -- 172.21.15.170:0/825872472 shutdown_connections 2024-03-29T11:14:47.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.404+0000 7ff5817fa640 1 -- 172.21.15.170:0/825872472 >> 172.21.15.170:0/825872472 conn(0x7ff59c0b35e0 msgr2=0x7ff59c13b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:47.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.405+0000 7ff5817fa640 1 -- 172.21.15.170:0/825872472 shutdown_connections 2024-03-29T11:14:47.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.405+0000 7ff5817fa640 1 -- 172.21.15.170:0/825872472 wait complete. 2024-03-29T11:14:47.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:47.406+0000 7ff5817fa640 1 librados: shutdown 2024-03-29T11:14:47.401 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:54.206 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:54.360 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.380+0000 7f8e4b97b640 10 monclient: get_monmap_and_config 2024-03-29T11:14:54.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.380+0000 7f8e4b97b640 10 monclient: build_initial_monmap 2024-03-29T11:14:54.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.380+0000 7f8e4b97b640 10 monclient: monmap: 2024-03-29T11:14:54.361 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:14:54.361 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:54.361 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:14:54.380809+0000 2024-03-29T11:14:54.361 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:14:54.380809+0000 2024-03-29T11:14:54.361 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:54.361 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:54.361 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:14:54.361 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:14:54.361 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:14:54.361 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:54.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.381+0000 7f8e4b97b640 1 Processor -- start 2024-03-29T11:14:54.361 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.382+0000 7f8e4b97b640 1 -- start start 2024-03-29T11:14:54.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.382+0000 7f8e4b97b640 10 monclient: init 2024-03-29T11:14:54.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.383+0000 7f8e4b97b640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:54.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.383+0000 7f8e4b97b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:54.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.383+0000 7f8e4b97b640 10 monclient(hunting): picked mon.noname-b con 0x7f8e440ad6c0 addr v1:172.21.15.170:6790/0 2024-03-29T11:14:54.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.383+0000 7f8e4b97b640 10 monclient(hunting): picked mon.noname-a con 0x7f8e440af6c0 addr v1:172.21.15.170:6789/0 2024-03-29T11:14:54.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.383+0000 7f8e4b97b640 10 monclient(hunting): picked mon.noname-c con 0x7f8e4414fdc0 addr v1:172.21.15.184:6789/0 2024-03-29T11:14:54.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.383+0000 7f8e4b97b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e440a94e0 con 0x7f8e440af6c0 2024-03-29T11:14:54.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.383+0000 7f8e4b97b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e440af3a0 con 0x7f8e4414fdc0 2024-03-29T11:14:54.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.383+0000 7f8e4b97b640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e440afb60 con 0x7f8e440ad6c0 2024-03-29T11:14:54.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.383+0000 7f8e4b97b640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:54.362 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.383+0000 7f8e4b97b640 10 monclient(hunting): authenticate will time out at 1075.694288s 2024-03-29T11:14:54.363 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.387+0000 7f8e49f18640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f8e4414fdc0 0x7f8e4415a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:52691/0 (socket says 172.21.15.170:52691) 2024-03-29T11:14:54.363 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.387+0000 7f8e49f18640 1 -- 172.21.15.170:0/2360790005 learned_addr learned my addr 172.21.15.170:0/2360790005 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:14:54.368 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 1 -- 172.21.15.170:0/2360790005 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833515387 0 0) 0x7f8e440af3a0 con 0x7f8e4414fdc0 2024-03-29T11:14:54.368 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:54.368 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:54.368 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:54.368 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:14:54.368 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 1 -- 172.21.15.170:0/2360790005 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e30003520 con 0x7f8e4414fdc0 2024-03-29T11:14:54.368 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 1 -- 172.21.15.170:0/2360790005 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 121417066 0 0) 0x7f8e30003520 con 0x7f8e4414fdc0 2024-03-29T11:14:54.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 1 -- 172.21.15.170:0/2360790005 >> v1:172.21.15.170:6790/0 conn(0x7f8e440ad6c0 legacy=0x7f8e4414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:54.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 1 -- 172.21.15.170:0/2360790005 >> v1:172.21.15.170:6789/0 conn(0x7f8e440af6c0 legacy=0x7f8e44152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:54.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:14:54.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 1 monclient: found mon.noname-c 2024-03-29T11:14:54.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:54.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:14:54.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 1 -- 172.21.15.170:0/2360790005 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e440aab80 con 0x7f8e4414fdc0 2024-03-29T11:14:54.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 10 monclient: _finish_auth 0 2024-03-29T11:14:54.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:14:54.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:54.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 1 -- 172.21.15.170:0/2360790005 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f8e340012f0 con 0x7f8e4414fdc0 2024-03-29T11:14:54.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:54.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:54.370 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 10 monclient: dump: 2024-03-29T11:14:54.370 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:54.370 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:54.370 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:54.370 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:54.370 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:54.370 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:54.370 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:54.370 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:54.370 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:54.370 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:54.370 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e4b97b640 5 monclient: authenticate success, global_id 4333 2024-03-29T11:14:54.370 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e4b97b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:54.370 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 1 -- 172.21.15.170:0/2360790005 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e34001520 con 0x7f8e4414fdc0 2024-03-29T11:14:54.370 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:54.371 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 1 -- 172.21.15.170:0/2360790005 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f8e340018b0 con 0x7f8e4414fdc0 2024-03-29T11:14:54.371 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:54.371 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:54.371 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e3bfff640 10 monclient: dump: 2024-03-29T11:14:54.371 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:54.371 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:54.371 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:54.371 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:54.372 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:54.372 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:54.372 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:54.372 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:54.372 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:54.373 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:54.373 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e4b97b640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:54.373 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.391+0000 7f8e4b97b640 4 set_mon_vals no callback set 2024-03-29T11:14:54.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.392+0000 7f8e4b97b640 10 monclient: shutdown 2024-03-29T11:14:54.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.392+0000 7f8e4b97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:54.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.392+0000 7f8e4b97b640 1 -- 172.21.15.170:0/2360790005 >> v1:172.21.15.184:6789/0 conn(0x7f8e4414fdc0 legacy=0x7f8e4415a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:54.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.393+0000 7f8e4b97b640 1 -- 172.21.15.170:0/2360790005 shutdown_connections 2024-03-29T11:14:54.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.393+0000 7f8e4b97b640 1 -- 172.21.15.170:0/2360790005 >> 172.21.15.170:0/2360790005 conn(0x7f8e440b35e0 msgr2=0x7f8e440b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:54.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.393+0000 7f8e4b97b640 1 -- 172.21.15.170:0/2360790005 shutdown_connections 2024-03-29T11:14:54.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.393+0000 7f8e4b97b640 1 -- 172.21.15.170:0/2360790005 wait complete. 2024-03-29T11:14:54.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.394+0000 7f8e4b97b640 10 monclient: build_initial_monmap 2024-03-29T11:14:54.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.394+0000 7f8e4b97b640 10 monclient: monmap: 2024-03-29T11:14:54.376 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:14:54.377 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:54.377 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:14:54.377 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:14:54.377 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:54.377 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:54.377 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:14:54.377 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:14:54.377 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:14:54.378 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:54.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.395+0000 7f8e4b97b640 1 librados: starting msgr at 2024-03-29T11:14:54.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.395+0000 7f8e4b97b640 1 librados: starting objecter 2024-03-29T11:14:54.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.395+0000 7f8e4b97b640 1 Processor -- start 2024-03-29T11:14:54.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.395+0000 7f8e4b97b640 1 -- start start 2024-03-29T11:14:54.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.395+0000 7f8e4b97b640 1 librados: setting wanted keys 2024-03-29T11:14:54.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.395+0000 7f8e4b97b640 1 librados: calling monclient init 2024-03-29T11:14:54.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.395+0000 7f8e4b97b640 10 monclient: init 2024-03-29T11:14:54.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.396+0000 7f8e4b97b640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:54.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.396+0000 7f8e4b97b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:14:54.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.396+0000 7f8e4b97b640 10 monclient(hunting): picked mon.noname-a con 0x7f8e440ad6c0 addr v1:172.21.15.170:6789/0 2024-03-29T11:14:54.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.396+0000 7f8e4b97b640 10 monclient(hunting): picked mon.noname-c con 0x7f8e440af6c0 addr v1:172.21.15.170:6790/0 2024-03-29T11:14:54.380 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.396+0000 7f8e4b97b640 10 monclient(hunting): picked mon.noname-b con 0x7f8e4414fdc0 addr v1:172.21.15.184:6789/0 2024-03-29T11:14:54.380 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.396+0000 7f8e4b97b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e44070b40 con 0x7f8e440ad6c0 2024-03-29T11:14:54.380 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.396+0000 7f8e4b97b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4406e8b0 con 0x7f8e4414fdc0 2024-03-29T11:14:54.380 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.396+0000 7f8e4b97b640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e441522b0 con 0x7f8e440af6c0 2024-03-29T11:14:54.380 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.396+0000 7f8e4b97b640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:54.380 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.396+0000 7f8e4b97b640 10 monclient(hunting): authenticate will time out at 1075.707323s 2024-03-29T11:14:54.391 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.404+0000 7f8e49717640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f8e440ad6c0 0x7f8e4408a410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:55919/0 (socket says 172.21.15.170:55919) 2024-03-29T11:14:54.392 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.404+0000 7f8e49717640 1 -- 172.21.15.170:0/3976341844 learned_addr learned my addr 172.21.15.170:0/3976341844 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:14:54.392 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.405+0000 7f8e39ffb640 10 client.?.objecter ms_handle_connect 0x7f8e440ad6c0 2024-03-29T11:14:54.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.405+0000 7f8e39ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:54.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.405+0000 7f8e39ffb640 10 client.?.objecter ms_handle_connect 0x7f8e4414fdc0 2024-03-29T11:14:54.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.405+0000 7f8e39ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:54.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.406+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013999708 0 0) 0x7f8e440afb60 con 0x7f8e440ad6c0 2024-03-29T11:14:54.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.406+0000 7f8e39ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:54.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.406+0000 7f8e39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:54.394 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.406+0000 7f8e39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:54.394 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.406+0000 7f8e39ffb640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:14:54.394 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.406+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e28004eb0 con 0x7f8e440ad6c0 2024-03-29T11:14:54.394 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.407+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767193837 0 0) 0x7f8e4406e8b0 con 0x7f8e4414fdc0 2024-03-29T11:14:54.394 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.407+0000 7f8e39ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:54.394 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.407+0000 7f8e39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:54.394 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.407+0000 7f8e39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:54.394 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.407+0000 7f8e39ffb640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:14:54.394 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.407+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e440afb60 con 0x7f8e4414fdc0 2024-03-29T11:14:54.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.408+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1486128467 0 0) 0x7f8e28004eb0 con 0x7f8e440ad6c0 2024-03-29T11:14:54.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.408+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e4406e8b0 con 0x7f8e440ad6c0 2024-03-29T11:14:54.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.408+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f8e400017b0 con 0x7f8e440ad6c0 2024-03-29T11:14:54.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.408+0000 7f8e39ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:54.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.408+0000 7f8e39ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:54.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.408+0000 7f8e39ffb640 10 monclient(hunting): dump: 2024-03-29T11:14:54.395 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:54.395 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:54.395 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:54.395 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:54.395 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:54.395 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:54.395 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:54.395 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:54.395 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:54.396 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:54.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.408+0000 7f8e39ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:54.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.408+0000 7f8e39ffb640 10 client.?.objecter ms_handle_connect 0x7f8e440af6c0 2024-03-29T11:14:54.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.408+0000 7f8e39ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:54.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.409+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1677214012 0 0) 0x7f8e440afb60 con 0x7f8e4414fdc0 2024-03-29T11:14:54.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.409+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e28004eb0 con 0x7f8e4414fdc0 2024-03-29T11:14:54.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.409+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f8e34001690 con 0x7f8e4414fdc0 2024-03-29T11:14:54.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.409+0000 7f8e39ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:54.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.409+0000 7f8e39ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:54.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.409+0000 7f8e39ffb640 10 monclient(hunting): dump: 2024-03-29T11:14:54.396 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:54.396 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:54.396 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:54.396 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:54.396 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.409+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2557657390 0 0) 0x7f8e28004eb0 con 0x7f8e4414fdc0 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.409+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 >> v1:172.21.15.170:6790/0 conn(0x7f8e440af6c0 legacy=0x7f8e441f8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.409+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 >> v1:172.21.15.170:6789/0 conn(0x7f8e440ad6c0 legacy=0x7f8e4408a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.409+0000 7f8e39ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.409+0000 7f8e39ffb640 1 monclient: found mon.b 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.409+0000 7f8e39ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.409+0000 7f8e39ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.409+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e44152440 con 0x7f8e4414fdc0 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.410+0000 7f8e39ffb640 10 monclient: _finish_auth 0 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.410+0000 7f8e39ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:14:54.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.410+0000 7f8e39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:54.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.410+0000 7f8e4b97b640 5 monclient: authenticate success, global_id 4336 2024-03-29T11:14:54.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.410+0000 7f8e4b97b640 10 monclient: _renew_subs 2024-03-29T11:14:54.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.410+0000 7f8e4b97b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:54.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.410+0000 7f8e4b97b640 1 -- 172.21.15.170:0/3976341844 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e440b2c50 con 0x7f8e4414fdc0 2024-03-29T11:14:54.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.410+0000 7f8e4b97b640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:54.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.410+0000 7f8e4b97b640 10 monclient: _renew_subs 2024-03-29T11:14:54.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.410+0000 7f8e4b97b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:54.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.410+0000 7f8e4b97b640 1 -- 172.21.15.170:0/3976341844 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e4408ad00 con 0x7f8e4414fdc0 2024-03-29T11:14:54.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.410+0000 7f8e4b97b640 1 librados: init done 2024-03-29T11:14:54.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.412+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e34001be0 con 0x7f8e4414fdc0 2024-03-29T11:14:54.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.412+0000 7f8e39ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:54.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.412+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f8e34001d50 con 0x7f8e4414fdc0 2024-03-29T11:14:54.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.412+0000 7f8e39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:54.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.412+0000 7f8e39ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:54.399 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.412+0000 7f8e39ffb640 10 monclient: dump: 2024-03-29T11:14:54.399 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:14:54.399 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:14:54.399 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:54.399 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:14:54.399 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:54.399 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:14:54.399 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:14:54.399 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:14:54.399 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:14:54.399 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:14:54.399 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.413+0000 7f8e3b7fe640 4 set_mon_vals no callback set 2024-03-29T11:14:54.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.415+0000 7f8e4b97b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:54.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.415+0000 7f8e4b97b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:54.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.415+0000 7f8e4b97b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:54.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.415+0000 7f8e4b97b640 1 -- 172.21.15.170:0/3976341844 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e24005080 con 0x7f8e4414fdc0 2024-03-29T11:14:54.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.416+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f8e3401ac40 con 0x7f8e4414fdc0 2024-03-29T11:14:54.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.417+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f8e340849c0 con 0x7f8e4414fdc0 2024-03-29T11:14:54.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.417+0000 7f8e39ffb640 10 client.4336.objecter ms_dispatch 0x7f8e44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:54.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.417+0000 7f8e39ffb640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:54.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.417+0000 7f8e39ffb640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:54.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.417+0000 7f8e39ffb640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-29T11:14:54.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.418+0000 7f8e49717640 10 monclient: get_auth_request con 0x7f8e2806df10 auth_method 0 2024-03-29T11:14:54.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.418+0000 7f8e39ffb640 10 client.4336.objecter ms_handle_connect 0x7f8e2806df10 2024-03-29T11:14:54.404 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.427+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e340502a0 con 0x7f8e4414fdc0 2024-03-29T11:14:54.404 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.427+0000 7f8e39ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:54.404 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.427+0000 7f8e39ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:54.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.756+0000 7f8e4b97b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:54.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.756+0000 7f8e4b97b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:54.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.756+0000 7f8e4b97b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:14:54.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.756+0000 7f8e4b97b640 1 -- 172.21.15.170:0/3976341844 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e24005330 con 0x7f8e4414fdc0 2024-03-29T11:14:54.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.759+0000 7f8e39ffb640 1 -- 172.21.15.170:0/3976341844 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f8e340521d0 con 0x7f8e4414fdc0 2024-03-29T11:14:54.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.759+0000 7f8e39ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:54.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.759+0000 7f8e39ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:54.738 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:14:54.738 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:54.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.765+0000 7f8e4b97b640 10 librados: watch_flush enter 2024-03-29T11:14:54.742 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.765+0000 7f8e4b97b640 10 librados: watch_flush exit 2024-03-29T11:14:54.742 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.765+0000 7f8e4b97b640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-29T11:14:54.742 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.765+0000 7f8e4b97b640 10 client.4336.objecter successfully canceled tick 2024-03-29T11:14:54.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.766+0000 7f8e4b97b640 1 -- 172.21.15.170:0/3976341844 >> v1:172.21.15.170:6816/3298994444 conn(0x7f8e2806df10 legacy=0x7f8e280703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:54.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.766+0000 7f8e4b97b640 10 monclient: shutdown 2024-03-29T11:14:54.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.766+0000 7f8e4b97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:54.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.766+0000 7f8e4b97b640 1 -- 172.21.15.170:0/3976341844 >> v1:172.21.15.184:6789/0 conn(0x7f8e4414fdc0 legacy=0x7f8e44200c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:54.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.766+0000 7f8e4b97b640 1 -- 172.21.15.170:0/3976341844 shutdown_connections 2024-03-29T11:14:54.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.766+0000 7f8e4b97b640 1 -- 172.21.15.170:0/3976341844 >> 172.21.15.170:0/3976341844 conn(0x7f8e440b35e0 msgr2=0x7f8e4414f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:54.744 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.766+0000 7f8e4b97b640 1 -- 172.21.15.170:0/3976341844 shutdown_connections 2024-03-29T11:14:54.744 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.766+0000 7f8e4b97b640 1 -- 172.21.15.170:0/3976341844 wait complete. 2024-03-29T11:14:54.744 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:14:54.766+0000 7f8e4b97b640 1 librados: shutdown 2024-03-29T11:14:54.762 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:01.569 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:01.725 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.745+0000 7f0ce1fd6640 10 monclient: get_monmap_and_config 2024-03-29T11:15:01.725 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.745+0000 7f0ce1fd6640 10 monclient: build_initial_monmap 2024-03-29T11:15:01.725 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.745+0000 7f0ce1fd6640 10 monclient: monmap: 2024-03-29T11:15:01.725 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:15:01.725 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:01.725 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:15:01.746746+0000 2024-03-29T11:15:01.726 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:15:01.746746+0000 2024-03-29T11:15:01.726 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:01.726 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:01.726 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:15:01.726 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:15:01.726 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:15:01.726 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:01.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.746+0000 7f0ce1fd6640 1 Processor -- start 2024-03-29T11:15:01.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.746+0000 7f0ce1fd6640 1 -- start start 2024-03-29T11:15:01.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.746+0000 7f0ce1fd6640 10 monclient: init 2024-03-29T11:15:01.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.747+0000 7f0ce1fd6640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:01.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.747+0000 7f0ce1fd6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:15:01.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.747+0000 7f0ce1fd6640 10 monclient(hunting): picked mon.noname-b con 0x7f0cd4006890 addr v1:172.21.15.170:6790/0 2024-03-29T11:15:01.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.747+0000 7f0ce1fd6640 10 monclient(hunting): picked mon.noname-c con 0x7f0cd4006c70 addr v1:172.21.15.184:6789/0 2024-03-29T11:15:01.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.747+0000 7f0ce1fd6640 10 monclient(hunting): picked mon.noname-a con 0x7f0cd402b4c0 addr v1:172.21.15.170:6789/0 2024-03-29T11:15:01.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.747+0000 7f0ce1fd6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd402c830 con 0x7f0cd402b4c0 2024-03-29T11:15:01.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.747+0000 7f0ce1fd6640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd402afa0 con 0x7f0cd4006c70 2024-03-29T11:15:01.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.747+0000 7f0ce1fd6640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd402aba0 con 0x7f0cd4006890 2024-03-29T11:15:01.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.747+0000 7f0ce1fd6640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:01.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.747+0000 7f0ce1fd6640 10 monclient(hunting): authenticate will time out at 1083.059761s 2024-03-29T11:15:01.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.751+0000 7f0ce0fd4640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f0cd4006890 0x7f0cd40c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:51903/0 (socket says 172.21.15.170:51903) 2024-03-29T11:15:01.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.751+0000 7f0ce0fd4640 1 -- 172.21.15.170:0/3351529587 learned_addr learned my addr 172.21.15.170:0/3351529587 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:15:01.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.752+0000 7f0cdb7fe640 1 -- 172.21.15.170:0/3351529587 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3729459395 0 0) 0x7f0cd402c830 con 0x7f0cd402b4c0 2024-03-29T11:15:01.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.752+0000 7f0cdb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:01.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.752+0000 7f0cdb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:01.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.752+0000 7f0cdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:01.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.752+0000 7f0cdb7fe640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:15:01.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.752+0000 7f0cdb7fe640 1 -- 172.21.15.170:0/3351529587 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ccc003520 con 0x7f0cd402b4c0 2024-03-29T11:15:01.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0cdb7fe640 1 -- 172.21.15.170:0/3351529587 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 71794095 0 0) 0x7f0ccc003520 con 0x7f0cd402b4c0 2024-03-29T11:15:01.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0cdb7fe640 1 -- 172.21.15.170:0/3351529587 >> v1:172.21.15.170:6790/0 conn(0x7f0cd4006890 legacy=0x7f0cd40c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:01.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0cdb7fe640 1 -- 172.21.15.170:0/3351529587 >> v1:172.21.15.184:6789/0 conn(0x7f0cd4006c70 legacy=0x7f0cd40c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:01.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0cdb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:15:01.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0cdb7fe640 1 monclient: found mon.noname-a 2024-03-29T11:15:01.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0cdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:01.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0cdb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:15:01.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0cdb7fe640 1 -- 172.21.15.170:0/3351529587 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd402ad30 con 0x7f0cd402b4c0 2024-03-29T11:15:01.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0cdb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:15:01.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0cdb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:15:01.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0cdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:01.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0ce1fd6640 5 monclient: authenticate success, global_id 4335 2024-03-29T11:15:01.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0ce1fd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:01.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0cdb7fe640 1 -- 172.21.15.170:0/3351529587 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f0cdc04f730 con 0x7f0cd402b4c0 2024-03-29T11:15:01.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0cdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:01.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0cdb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:01.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0cdb7fe640 10 monclient: dump: 2024-03-29T11:15:01.737 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:01.737 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:01.737 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:01.737 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:01.737 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:01.737 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:01.737 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:01.737 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:01.737 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:01.737 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:01.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0ce1fd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:01.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0cdb7fe640 1 -- 172.21.15.170:0/3351529587 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cdc075f10 con 0x7f0cd402b4c0 2024-03-29T11:15:01.740 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0cdb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:01.740 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0ce1fd6640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:01.740 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.755+0000 7f0ce1fd6640 4 set_mon_vals no callback set 2024-03-29T11:15:01.740 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.756+0000 7f0cdb7fe640 1 -- 172.21.15.170:0/3351529587 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f0cdc0758b0 con 0x7f0cd402b4c0 2024-03-29T11:15:01.740 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.756+0000 7f0cdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:01.740 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.756+0000 7f0cdb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:01.740 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.756+0000 7f0cdb7fe640 10 monclient: dump: 2024-03-29T11:15:01.740 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:01.740 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:01.740 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:01.740 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:01.740 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:01.740 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:01.740 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:01.741 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:01.741 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:01.741 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:01.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.756+0000 7f0ce1fd6640 10 monclient: shutdown 2024-03-29T11:15:01.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.757+0000 7f0ce1fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:01.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.757+0000 7f0ce1fd6640 1 -- 172.21.15.170:0/3351529587 >> v1:172.21.15.170:6789/0 conn(0x7f0cd402b4c0 legacy=0x7f0cd40cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:01.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.757+0000 7f0ce1fd6640 1 -- 172.21.15.170:0/3351529587 shutdown_connections 2024-03-29T11:15:01.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.757+0000 7f0ce1fd6640 1 -- 172.21.15.170:0/3351529587 >> 172.21.15.170:0/3351529587 conn(0x7f0cd40058f0 msgr2=0x7f0cd4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:01.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.760+0000 7f0ce1fd6640 1 -- 172.21.15.170:0/3351529587 shutdown_connections 2024-03-29T11:15:01.745 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.760+0000 7f0ce1fd6640 1 -- 172.21.15.170:0/3351529587 wait complete. 2024-03-29T11:15:01.745 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.761+0000 7f0ce1fd6640 10 monclient: build_initial_monmap 2024-03-29T11:15:01.745 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.761+0000 7f0ce1fd6640 10 monclient: monmap: 2024-03-29T11:15:01.745 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:15:01.745 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:01.746 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:15:01.746 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:15:01.746 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:01.746 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:01.746 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:15:01.746 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:15:01.746 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:15:01.746 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:01.747 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.762+0000 7f0ce1fd6640 1 librados: starting msgr at 2024-03-29T11:15:01.747 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.762+0000 7f0ce1fd6640 1 librados: starting objecter 2024-03-29T11:15:01.747 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.762+0000 7f0ce1fd6640 1 Processor -- start 2024-03-29T11:15:01.748 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.762+0000 7f0ce1fd6640 1 -- start start 2024-03-29T11:15:01.749 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.762+0000 7f0ce1fd6640 1 librados: setting wanted keys 2024-03-29T11:15:01.749 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.762+0000 7f0ce1fd6640 1 librados: calling monclient init 2024-03-29T11:15:01.749 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.762+0000 7f0ce1fd6640 10 monclient: init 2024-03-29T11:15:01.749 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.763+0000 7f0ce1fd6640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:01.749 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.763+0000 7f0ce1fd6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:15:01.759 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.763+0000 7f0ce1fd6640 10 monclient(hunting): picked mon.noname-b con 0x7f0cd4006890 addr v1:172.21.15.184:6789/0 2024-03-29T11:15:01.760 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.763+0000 7f0ce1fd6640 10 monclient(hunting): picked mon.noname-a con 0x7f0cd402b4c0 addr v1:172.21.15.170:6789/0 2024-03-29T11:15:01.760 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.763+0000 7f0ce1fd6640 10 monclient(hunting): picked mon.noname-c con 0x7f0cd402df10 addr v1:172.21.15.170:6790/0 2024-03-29T11:15:01.760 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.763+0000 7f0ce1fd6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd402f5a0 con 0x7f0cd402b4c0 2024-03-29T11:15:01.760 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.763+0000 7f0ce1fd6640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd402f880 con 0x7f0cd4006890 2024-03-29T11:15:01.760 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.763+0000 7f0ce1fd6640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd402d220 con 0x7f0cd402df10 2024-03-29T11:15:01.760 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.763+0000 7f0ce1fd6640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:01.760 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.763+0000 7f0ce1fd6640 10 monclient(hunting): authenticate will time out at 1083.075706s 2024-03-29T11:15:01.760 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.768+0000 7f0ce17d5640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f0cd402df10 0x7f0cd416be40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:51991/0 (socket says 172.21.15.170:51991) 2024-03-29T11:15:01.760 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.769+0000 7f0ce17d5640 1 -- 172.21.15.170:0/2977441136 learned_addr learned my addr 172.21.15.170:0/2977441136 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:15:01.760 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.769+0000 7f0cd97fa640 10 client.?.objecter ms_handle_connect 0x7f0cd402df10 2024-03-29T11:15:01.760 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.769+0000 7f0cd97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:01.760 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.769+0000 7f0cd97fa640 1 -- 172.21.15.170:0/2977441136 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747286897 0 0) 0x7f0cd402d220 con 0x7f0cd402df10 2024-03-29T11:15:01.761 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.769+0000 7f0cd97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:15:01.761 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.769+0000 7f0cd97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:01.761 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.769+0000 7f0cd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:01.761 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.769+0000 7f0cd97fa640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:15:01.761 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.769+0000 7f0cd97fa640 1 -- 172.21.15.170:0/2977441136 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cc0004eb0 con 0x7f0cd402df10 2024-03-29T11:15:01.761 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.770+0000 7f0cd97fa640 1 -- 172.21.15.170:0/2977441136 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 329522528 0 0) 0x7f0cc0004eb0 con 0x7f0cd402df10 2024-03-29T11:15:01.761 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.770+0000 7f0cd97fa640 1 -- 172.21.15.170:0/2977441136 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cd402d220 con 0x7f0cd402df10 2024-03-29T11:15:01.761 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.770+0000 7f0cd97fa640 1 -- 172.21.15.170:0/2977441136 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f0cdc04f400 con 0x7f0cd402df10 2024-03-29T11:15:01.761 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.770+0000 7f0cd97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:01.761 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.770+0000 7f0cd97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:01.761 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.770+0000 7f0cd97fa640 10 monclient(hunting): dump: 2024-03-29T11:15:01.761 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:01.761 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:01.761 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:01.762 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:01.762 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:01.762 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:01.762 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:01.762 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:01.762 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:01.762 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:01.762 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.770+0000 7f0cd97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:01.762 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.770+0000 7f0cd97fa640 1 -- 172.21.15.170:0/2977441136 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2618327560 0 0) 0x7f0cd402d220 con 0x7f0cd402df10 2024-03-29T11:15:01.762 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.770+0000 7f0cd97fa640 1 -- 172.21.15.170:0/2977441136 >> v1:172.21.15.184:6789/0 conn(0x7f0cd4006890 legacy=0x7f0cd40b8900 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:15:01.762 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.770+0000 7f0cd97fa640 1 -- 172.21.15.170:0/2977441136 >> v1:172.21.15.170:6789/0 conn(0x7f0cd402b4c0 legacy=0x7f0cd40b9010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:15:01.762 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.771+0000 7f0cd97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:15:01.762 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.771+0000 7f0cd97fa640 1 monclient: found mon.c 2024-03-29T11:15:01.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.771+0000 7f0cd97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:01.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.771+0000 7f0cd97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:15:01.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.771+0000 7f0cd97fa640 1 -- 172.21.15.170:0/2977441136 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd40b52f0 con 0x7f0cd402df10 2024-03-29T11:15:01.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.771+0000 7f0cd97fa640 10 monclient: _finish_auth 0 2024-03-29T11:15:01.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.771+0000 7f0cd97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:15:01.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.771+0000 7f0cd97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:01.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.771+0000 7f0ce1fd6640 5 monclient: authenticate success, global_id 4364 2024-03-29T11:15:01.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.771+0000 7f0ce1fd6640 10 monclient: _renew_subs 2024-03-29T11:15:01.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.771+0000 7f0ce1fd6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:15:01.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.771+0000 7f0ce1fd6640 1 -- 172.21.15.170:0/2977441136 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd416d560 con 0x7f0cd402df10 2024-03-29T11:15:01.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.771+0000 7f0ce1fd6640 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:01.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.771+0000 7f0ce1fd6640 10 monclient: _renew_subs 2024-03-29T11:15:01.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.771+0000 7f0ce1fd6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:15:01.763 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.771+0000 7f0ce1fd6640 1 -- 172.21.15.170:0/2977441136 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd416d8d0 con 0x7f0cd402df10 2024-03-29T11:15:01.764 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.771+0000 7f0ce1fd6640 1 librados: init done 2024-03-29T11:15:01.764 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.772+0000 7f0cd97fa640 1 -- 172.21.15.170:0/2977441136 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cdc04fdd0 con 0x7f0cd402df10 2024-03-29T11:15:01.764 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.772+0000 7f0cd97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:01.764 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.772+0000 7f0cd97fa640 1 -- 172.21.15.170:0/2977441136 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f0cdc067bf0 con 0x7f0cd402df10 2024-03-29T11:15:01.764 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.772+0000 7f0cd97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:01.764 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.772+0000 7f0cd97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:01.764 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.772+0000 7f0cd97fa640 10 monclient: dump: 2024-03-29T11:15:01.764 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:01.764 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:01.764 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:01.764 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:01.764 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:01.764 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:01.764 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:01.765 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:01.765 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:01.765 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:01.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.773+0000 7f0cdaffd640 4 set_mon_vals no callback set 2024-03-29T11:15:01.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.774+0000 7f0cd97fa640 1 -- 172.21.15.170:0/2977441136 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f0cdc067dd0 con 0x7f0cd402df10 2024-03-29T11:15:01.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.774+0000 7f0cd97fa640 1 -- 172.21.15.170:0/2977441136 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f0cdc0ff7c0 con 0x7f0cd402df10 2024-03-29T11:15:01.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.774+0000 7f0cd97fa640 10 client.4364.objecter ms_dispatch 0x7f0cdc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:01.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.775+0000 7f0cd97fa640 3 client.4364.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:01.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.775+0000 7f0ce1fd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:01.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.775+0000 7f0ce1fd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:01.767 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.775+0000 7f0ce1fd6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:15:01.767 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.775+0000 7f0ce1fd6640 1 -- 172.21.15.170:0/2977441136 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cbc005080 con 0x7f0cd402df10 2024-03-29T11:15:01.767 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.775+0000 7f0ce0fd4640 10 monclient: get_auth_request con 0x7f0cc006da10 auth_method 0 2024-03-29T11:15:01.767 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.775+0000 7f0cd97fa640 3 client.4364.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:01.767 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.775+0000 7f0cd97fa640 20 client.4364.objecter dump_active .. 0 homeless 2024-03-29T11:15:01.767 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.779+0000 7f0cd97fa640 10 client.4364.objecter ms_handle_connect 0x7f0cc006da10 2024-03-29T11:15:01.767 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.784+0000 7f0cd97fa640 1 -- 172.21.15.170:0/2977441136 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0cdc0cb0a0 con 0x7f0cd402df10 2024-03-29T11:15:01.768 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.784+0000 7f0cd97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:01.768 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:01.784+0000 7f0cd97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:02.081 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.103+0000 7f0ce1fd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:02.082 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.103+0000 7f0ce1fd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:02.082 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.103+0000 7f0ce1fd6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:15:02.082 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.103+0000 7f0ce1fd6640 1 -- 172.21.15.170:0/2977441136 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cbc005330 con 0x7f0cd402df10 2024-03-29T11:15:02.084 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.106+0000 7f0cd97fa640 1 -- 172.21.15.170:0/2977441136 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f0cdc0ccfd0 con 0x7f0cd402df10 2024-03-29T11:15:02.084 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.106+0000 7f0cd97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:02.084 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.106+0000 7f0cd97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:02.084 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:15:02.085 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:02.088 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.109+0000 7f0ce1fd6640 10 librados: watch_flush enter 2024-03-29T11:15:02.088 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.110+0000 7f0ce1fd6640 10 librados: watch_flush exit 2024-03-29T11:15:02.088 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.110+0000 7f0ce1fd6640 20 client.4364.objecter shutdown clearing up homeless session... 2024-03-29T11:15:02.088 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.110+0000 7f0ce1fd6640 10 client.4364.objecter successfully canceled tick 2024-03-29T11:15:02.088 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.110+0000 7f0ce1fd6640 1 -- 172.21.15.170:0/2977441136 >> v1:172.21.15.170:6816/3298994444 conn(0x7f0cc006da10 legacy=0x7f0cc006fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:02.088 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.110+0000 7f0ce1fd6640 10 monclient: shutdown 2024-03-29T11:15:02.089 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.110+0000 7f0ce1fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:02.089 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.110+0000 7f0ce1fd6640 1 -- 172.21.15.170:0/2977441136 >> v1:172.21.15.170:6790/0 conn(0x7f0cd402df10 legacy=0x7f0cd416be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:02.089 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.110+0000 7f0ce1fd6640 1 -- 172.21.15.170:0/2977441136 shutdown_connections 2024-03-29T11:15:02.089 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.110+0000 7f0ce1fd6640 1 -- 172.21.15.170:0/2977441136 >> 172.21.15.170:0/2977441136 conn(0x7f0cd40058f0 msgr2=0x7f0cd4038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:02.090 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.111+0000 7f0ce1fd6640 1 -- 172.21.15.170:0/2977441136 shutdown_connections 2024-03-29T11:15:02.090 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.111+0000 7f0ce1fd6640 1 -- 172.21.15.170:0/2977441136 wait complete. 2024-03-29T11:15:02.090 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:02.111+0000 7f0ce1fd6640 1 librados: shutdown 2024-03-29T11:15:02.109 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:08.917 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:09.070 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.091+0000 7f59bff47640 10 monclient: get_monmap_and_config 2024-03-29T11:15:09.071 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.091+0000 7f59bff47640 10 monclient: build_initial_monmap 2024-03-29T11:15:09.071 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.092+0000 7f59bff47640 10 monclient: monmap: 2024-03-29T11:15:09.071 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:15:09.071 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:09.071 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:15:09.093443+0000 2024-03-29T11:15:09.071 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:15:09.093443+0000 2024-03-29T11:15:09.071 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:09.071 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:09.072 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:15:09.072 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:15:09.072 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:15:09.072 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:09.072 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.092+0000 7f59bff47640 1 Processor -- start 2024-03-29T11:15:09.072 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.093+0000 7f59bff47640 1 -- start start 2024-03-29T11:15:09.075 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.093+0000 7f59bff47640 10 monclient: init 2024-03-29T11:15:09.075 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.093+0000 7f59bff47640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:09.075 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.093+0000 7f59bff47640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:09.075 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.094+0000 7f59bff47640 10 monclient(hunting): picked mon.noname-a con 0x7f59b80b0770 addr v1:172.21.15.170:6789/0 2024-03-29T11:15:09.075 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.094+0000 7f59bff47640 10 monclient(hunting): picked mon.noname-c con 0x7f59b814a700 addr v1:172.21.15.184:6789/0 2024-03-29T11:15:09.075 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.094+0000 7f59bff47640 10 monclient(hunting): picked mon.noname-b con 0x7f59b80ac5a0 addr v1:172.21.15.170:6790/0 2024-03-29T11:15:09.075 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.094+0000 7f59bff47640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b80aba70 con 0x7f59b80b0770 2024-03-29T11:15:09.075 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.094+0000 7f59bff47640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b80aecb0 con 0x7f59b814a700 2024-03-29T11:15:09.076 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.094+0000 7f59bff47640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b80af150 con 0x7f59b80ac5a0 2024-03-29T11:15:09.076 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.094+0000 7f59bff47640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:09.076 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.094+0000 7f59bff47640 10 monclient(hunting): authenticate will time out at 1090.406089s 2024-03-29T11:15:09.078 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.100+0000 7f59bd4e2640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f59b814a700 0x7f59b80abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:46827/0 (socket says 172.21.15.170:46827) 2024-03-29T11:15:09.078 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.100+0000 7f59bd4e2640 1 -- 172.21.15.170:0/3593377389 learned_addr learned my addr 172.21.15.170:0/3593377389 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:15:09.080 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.101+0000 7f59bcce1640 1 -- 172.21.15.170:0/3593377389 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873993062 0 0) 0x7f59b80af150 con 0x7f59b80ac5a0 2024-03-29T11:15:09.080 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.101+0000 7f59bcce1640 10 monclient(hunting): _negotiate 2024-03-29T11:15:09.080 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.101+0000 7f59bcce1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:09.080 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.101+0000 7f59bcce1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:09.081 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.101+0000 7f59bcce1640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:15:09.081 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.101+0000 7f59bcce1640 1 -- 172.21.15.170:0/3593377389 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59a4003520 con 0x7f59b80ac5a0 2024-03-29T11:15:09.086 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.103+0000 7f59bcce1640 1 -- 172.21.15.170:0/3593377389 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3981915687 0 0) 0x7f59a4003520 con 0x7f59b80ac5a0 2024-03-29T11:15:09.086 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.103+0000 7f59bcce1640 1 -- 172.21.15.170:0/3593377389 >> v1:172.21.15.184:6789/0 conn(0x7f59b814a700 legacy=0x7f59b80abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:09.086 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bcce1640 1 -- 172.21.15.170:0/3593377389 >> v1:172.21.15.170:6789/0 conn(0x7f59b80b0770 legacy=0x7f59b8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:09.086 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bcce1640 10 monclient: _finish_hunting 0 2024-03-29T11:15:09.087 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bcce1640 1 monclient: found mon.noname-b 2024-03-29T11:15:09.087 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bcce1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:09.087 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bcce1640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:15:09.087 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bcce1640 1 -- 172.21.15.170:0/3593377389 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b80ac9d0 con 0x7f59b80ac5a0 2024-03-29T11:15:09.087 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bcce1640 10 monclient: _finish_auth 0 2024-03-29T11:15:09.088 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bcce1640 10 monclient: _check_auth_tickets 2024-03-29T11:15:09.088 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bcce1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:09.088 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bff47640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:15:09.088 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bff47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:09.088 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bcce1640 1 -- 172.21.15.170:0/3593377389 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f59b0001a10 con 0x7f59b80ac5a0 2024-03-29T11:15:09.088 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bcce1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:09.088 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bcce1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:09.088 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bcce1640 10 monclient: dump: 2024-03-29T11:15:09.089 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:09.089 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:09.089 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:09.089 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:09.089 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:09.089 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:09.089 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:09.089 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:09.089 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:09.089 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:09.089 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bff47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:09.089 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bcce1640 1 -- 172.21.15.170:0/3593377389 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59b0001b70 con 0x7f59b80ac5a0 2024-03-29T11:15:09.090 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bcce1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:09.090 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bff47640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:09.090 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bff47640 4 set_mon_vals no callback set 2024-03-29T11:15:09.090 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.104+0000 7f59bcce1640 1 -- 172.21.15.170:0/3593377389 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f59b0001fc0 con 0x7f59b80ac5a0 2024-03-29T11:15:09.090 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.105+0000 7f59bcce1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:09.090 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.105+0000 7f59bcce1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:09.090 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.105+0000 7f59bcce1640 10 monclient: dump: 2024-03-29T11:15:09.090 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:09.091 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:09.091 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:09.091 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:09.091 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:09.091 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:09.091 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:09.091 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:09.091 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:09.091 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:09.091 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.105+0000 7f59bff47640 10 monclient: shutdown 2024-03-29T11:15:09.091 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.105+0000 7f59bff47640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:09.091 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.105+0000 7f59bff47640 1 -- 172.21.15.170:0/3593377389 >> v1:172.21.15.170:6790/0 conn(0x7f59b80ac5a0 legacy=0x7f59b8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:09.091 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.105+0000 7f59bff47640 1 -- 172.21.15.170:0/3593377389 shutdown_connections 2024-03-29T11:15:09.092 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.105+0000 7f59bff47640 1 -- 172.21.15.170:0/3593377389 >> 172.21.15.170:0/3593377389 conn(0x7f59b806b660 msgr2=0x7f59b806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:09.092 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.105+0000 7f59bff47640 1 -- 172.21.15.170:0/3593377389 shutdown_connections 2024-03-29T11:15:09.092 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.105+0000 7f59bff47640 1 -- 172.21.15.170:0/3593377389 wait complete. 2024-03-29T11:15:09.094 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.106+0000 7f59bff47640 10 monclient: build_initial_monmap 2024-03-29T11:15:09.094 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.106+0000 7f59bff47640 10 monclient: monmap: 2024-03-29T11:15:09.094 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:15:09.095 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:09.095 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:15:09.095 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:15:09.095 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:09.095 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:09.095 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:15:09.095 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:15:09.095 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:15:09.095 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:09.095 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.106+0000 7f59bff47640 1 librados: starting msgr at 2024-03-29T11:15:09.095 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.106+0000 7f59bff47640 1 librados: starting objecter 2024-03-29T11:15:09.095 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.107+0000 7f59bff47640 1 Processor -- start 2024-03-29T11:15:09.095 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.107+0000 7f59bff47640 1 -- start start 2024-03-29T11:15:09.095 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.107+0000 7f59bff47640 1 librados: setting wanted keys 2024-03-29T11:15:09.095 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.107+0000 7f59bff47640 1 librados: calling monclient init 2024-03-29T11:15:09.095 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.107+0000 7f59bff47640 10 monclient: init 2024-03-29T11:15:09.096 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.108+0000 7f59bff47640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:09.096 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.108+0000 7f59bff47640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:09.096 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.108+0000 7f59bff47640 10 monclient(hunting): picked mon.noname-a con 0x7f59b80ac5a0 addr v1:172.21.15.170:6789/0 2024-03-29T11:15:09.096 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.108+0000 7f59bff47640 10 monclient(hunting): picked mon.noname-b con 0x7f59b80b0770 addr v1:172.21.15.184:6789/0 2024-03-29T11:15:09.096 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.108+0000 7f59bff47640 10 monclient(hunting): picked mon.noname-c con 0x7f59b814a700 addr v1:172.21.15.170:6790/0 2024-03-29T11:15:09.096 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.108+0000 7f59bff47640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b812a130 con 0x7f59b80ac5a0 2024-03-29T11:15:09.096 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.108+0000 7f59bff47640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b8127c30 con 0x7f59b80b0770 2024-03-29T11:15:09.096 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.108+0000 7f59bff47640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b81df490 con 0x7f59b814a700 2024-03-29T11:15:09.096 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.108+0000 7f59bff47640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:09.096 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.108+0000 7f59bff47640 10 monclient(hunting): authenticate will time out at 1090.420148s 2024-03-29T11:15:09.096 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.115+0000 7f59bdce3640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f59b80ac5a0 0x7f59b8146c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:38617/0 (socket says 172.21.15.170:38617) 2024-03-29T11:15:09.096 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.115+0000 7f59bdce3640 1 -- 172.21.15.170:0/537000444 learned_addr learned my addr 172.21.15.170:0/537000444 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:15:09.096 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.115+0000 7f59ae7fc640 10 client.?.objecter ms_handle_connect 0x7f59b80ac5a0 2024-03-29T11:15:09.096 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.116+0000 7f59ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:09.098 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.118+0000 7f59ae7fc640 1 -- 172.21.15.170:0/537000444 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 188215495 0 0) 0x7f59b812a130 con 0x7f59b80ac5a0 2024-03-29T11:15:09.098 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.118+0000 7f59ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:09.098 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.118+0000 7f59ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:09.098 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.118+0000 7f59ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:09.098 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.118+0000 7f59ae7fc640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:15:09.099 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.119+0000 7f59ae7fc640 1 -- 172.21.15.170:0/537000444 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f599c004eb0 con 0x7f59b80ac5a0 2024-03-29T11:15:09.099 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.119+0000 7f59ae7fc640 10 client.?.objecter ms_handle_connect 0x7f59b80b0770 2024-03-29T11:15:09.099 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.119+0000 7f59ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:09.099 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.119+0000 7f59ae7fc640 1 -- 172.21.15.170:0/537000444 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254713256 0 0) 0x7f59b8127c30 con 0x7f59b80b0770 2024-03-29T11:15:09.100 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.119+0000 7f59ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:09.100 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.119+0000 7f59ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:09.100 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.119+0000 7f59ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:09.100 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.119+0000 7f59ae7fc640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:15:09.100 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.119+0000 7f59ae7fc640 1 -- 172.21.15.170:0/537000444 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b812a130 con 0x7f59b80b0770 2024-03-29T11:15:09.104 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.122+0000 7f59ae7fc640 1 -- 172.21.15.170:0/537000444 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1860677288 0 0) 0x7f59b812a130 con 0x7f59b80b0770 2024-03-29T11:15:09.104 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.122+0000 7f59ae7fc640 1 -- 172.21.15.170:0/537000444 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59b8127c30 con 0x7f59b80b0770 2024-03-29T11:15:09.104 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.122+0000 7f59ae7fc640 1 -- 172.21.15.170:0/537000444 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f59a8001810 con 0x7f59b80b0770 2024-03-29T11:15:09.104 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.122+0000 7f59ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:09.104 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.122+0000 7f59ae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:09.104 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.122+0000 7f59ae7fc640 10 monclient(hunting): dump: 2024-03-29T11:15:09.104 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:09.104 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:09.104 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:09.104 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:09.104 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:09.104 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:09.105 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:09.105 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:09.105 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:09.105 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:09.105 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.122+0000 7f59ae7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:09.105 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.122+0000 7f59ae7fc640 1 -- 172.21.15.170:0/537000444 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3450624557 0 0) 0x7f59b8127c30 con 0x7f59b80b0770 2024-03-29T11:15:09.105 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.123+0000 7f59ae7fc640 1 -- 172.21.15.170:0/537000444 >> v1:172.21.15.170:6790/0 conn(0x7f59b814a700 legacy=0x7f59b8128860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:09.105 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.123+0000 7f59ae7fc640 1 -- 172.21.15.170:0/537000444 >> v1:172.21.15.170:6789/0 conn(0x7f59b80ac5a0 legacy=0x7f59b8146c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:09.105 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.123+0000 7f59ae7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:15:09.105 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.123+0000 7f59ae7fc640 1 monclient: found mon.b 2024-03-29T11:15:09.105 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.123+0000 7f59ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:09.105 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.123+0000 7f59ae7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:15:09.105 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.123+0000 7f59ae7fc640 1 -- 172.21.15.170:0/537000444 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b81df620 con 0x7f59b80b0770 2024-03-29T11:15:09.105 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.123+0000 7f59ae7fc640 10 monclient: _finish_auth 0 2024-03-29T11:15:09.105 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.123+0000 7f59ae7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:15:09.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.123+0000 7f59ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:09.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.123+0000 7f59bff47640 5 monclient: authenticate success, global_id 4342 2024-03-29T11:15:09.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.123+0000 7f59bff47640 10 monclient: _renew_subs 2024-03-29T11:15:09.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.123+0000 7f59bff47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:15:09.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.123+0000 7f59bff47640 1 -- 172.21.15.170:0/537000444 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b8147830 con 0x7f59b80b0770 2024-03-29T11:15:09.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.123+0000 7f59bff47640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:09.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.123+0000 7f59bff47640 10 monclient: _renew_subs 2024-03-29T11:15:09.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.124+0000 7f59bff47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:15:09.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.124+0000 7f59bff47640 1 -- 172.21.15.170:0/537000444 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b8147510 con 0x7f59b80b0770 2024-03-29T11:15:09.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.124+0000 7f59bff47640 1 librados: init done 2024-03-29T11:15:09.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.125+0000 7f59ae7fc640 1 -- 172.21.15.170:0/537000444 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59a8001d00 con 0x7f59b80b0770 2024-03-29T11:15:09.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.125+0000 7f59bff47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:09.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.125+0000 7f59bff47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:09.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.125+0000 7f59bff47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:15:09.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.125+0000 7f59bff47640 1 -- 172.21.15.170:0/537000444 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b80ab560 con 0x7f59b80b0770 2024-03-29T11:15:09.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.126+0000 7f59ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:09.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.126+0000 7f59ae7fc640 1 -- 172.21.15.170:0/537000444 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f59a8004980 con 0x7f59b80b0770 2024-03-29T11:15:09.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.126+0000 7f59ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:09.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.126+0000 7f59ae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:09.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.127+0000 7f59ae7fc640 10 monclient: dump: 2024-03-29T11:15:09.109 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:09.109 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:09.109 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:09.110 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:09.110 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:09.110 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:09.110 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:09.110 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:09.110 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:09.110 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:09.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.126+0000 7f59affff640 4 set_mon_vals no callback set 2024-03-29T11:15:09.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.128+0000 7f59ae7fc640 1 -- 172.21.15.170:0/537000444 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f59a8004ba0 con 0x7f59b80b0770 2024-03-29T11:15:09.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.128+0000 7f59bdce3640 10 monclient: get_auth_request con 0x7f599c06dd70 auth_method 0 2024-03-29T11:15:09.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.129+0000 7f59ae7fc640 10 client.4342.objecter ms_handle_connect 0x7f599c06dd70 2024-03-29T11:15:09.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.129+0000 7f59ae7fc640 1 -- 172.21.15.170:0/537000444 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f59a8084980 con 0x7f59b80b0770 2024-03-29T11:15:09.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.129+0000 7f59ae7fc640 10 client.4342.objecter ms_dispatch 0x7f59b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:09.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.129+0000 7f59ae7fc640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:09.111 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.129+0000 7f59ae7fc640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:09.111 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.129+0000 7f59ae7fc640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-29T11:15:09.112 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.134+0000 7f59ae7fc640 1 -- 172.21.15.170:0/537000444 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f59a80502e0 con 0x7f59b80b0770 2024-03-29T11:15:09.112 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.134+0000 7f59ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:09.112 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.134+0000 7f59ae7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:09.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.454+0000 7f59bff47640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:09.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.454+0000 7f59bff47640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:09.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.454+0000 7f59bff47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:15:09.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.454+0000 7f59bff47640 1 -- 172.21.15.170:0/537000444 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59b81479c0 con 0x7f59b80b0770 2024-03-29T11:15:09.439 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.461+0000 7f59ae7fc640 1 -- 172.21.15.170:0/537000444 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f59a8052210 con 0x7f59b80b0770 2024-03-29T11:15:09.439 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.461+0000 7f59ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:09.440 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.461+0000 7f59ae7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:09.440 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:15:09.440 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:09.444 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.466+0000 7f59bff47640 10 librados: watch_flush enter 2024-03-29T11:15:09.444 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.466+0000 7f59bff47640 10 librados: watch_flush exit 2024-03-29T11:15:09.444 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.466+0000 7f59bff47640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-29T11:15:09.444 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.466+0000 7f59bff47640 10 client.4342.objecter successfully canceled tick 2024-03-29T11:15:09.447 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.469+0000 7f59bff47640 1 -- 172.21.15.170:0/537000444 >> v1:172.21.15.170:6816/3298994444 conn(0x7f599c06dd70 legacy=0x7f599c070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:09.447 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.469+0000 7f59bff47640 10 monclient: shutdown 2024-03-29T11:15:09.447 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.469+0000 7f59bff47640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:09.447 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.469+0000 7f59bff47640 1 -- 172.21.15.170:0/537000444 >> v1:172.21.15.184:6789/0 conn(0x7f59b80b0770 legacy=0x7f59b8128150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:09.447 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.469+0000 7f59bff47640 1 -- 172.21.15.170:0/537000444 shutdown_connections 2024-03-29T11:15:09.447 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.469+0000 7f59bff47640 1 -- 172.21.15.170:0/537000444 >> 172.21.15.170:0/537000444 conn(0x7f59b806b660 msgr2=0x7f59b80b2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:09.448 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.470+0000 7f59bff47640 1 -- 172.21.15.170:0/537000444 shutdown_connections 2024-03-29T11:15:09.448 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.470+0000 7f59bff47640 1 -- 172.21.15.170:0/537000444 wait complete. 2024-03-29T11:15:09.448 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:09.470+0000 7f59bff47640 1 librados: shutdown 2024-03-29T11:15:09.469 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:16.277 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:16.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.450+0000 7f02ea2e5640 10 monclient: get_monmap_and_config 2024-03-29T11:15:16.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.450+0000 7f02ea2e5640 10 monclient: build_initial_monmap 2024-03-29T11:15:16.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.450+0000 7f02ea2e5640 10 monclient: monmap: 2024-03-29T11:15:16.428 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:15:16.428 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:16.428 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:15:16.451750+0000 2024-03-29T11:15:16.428 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:15:16.451750+0000 2024-03-29T11:15:16.429 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:16.429 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:16.429 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:15:16.429 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:15:16.429 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:15:16.429 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:16.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.452+0000 7f02ea2e5640 1 Processor -- start 2024-03-29T11:15:16.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.452+0000 7f02ea2e5640 1 -- start start 2024-03-29T11:15:16.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.452+0000 7f02ea2e5640 10 monclient: init 2024-03-29T11:15:16.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.453+0000 7f02ea2e5640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:16.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.453+0000 7f02ea2e5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:15:16.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.453+0000 7f02ea2e5640 10 monclient(hunting): picked mon.noname-c con 0x7f02dc006e30 addr v1:172.21.15.184:6789/0 2024-03-29T11:15:16.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.453+0000 7f02ea2e5640 10 monclient(hunting): picked mon.noname-a con 0x7f02dc035e00 addr v1:172.21.15.170:6789/0 2024-03-29T11:15:16.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.453+0000 7f02ea2e5640 10 monclient(hunting): picked mon.noname-b con 0x7f02dc0c9670 addr v1:172.21.15.170:6790/0 2024-03-29T11:15:16.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.453+0000 7f02ea2e5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02dc035970 con 0x7f02dc035e00 2024-03-29T11:15:16.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.453+0000 7f02ea2e5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02dc036280 con 0x7f02dc006e30 2024-03-29T11:15:16.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.453+0000 7f02ea2e5640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02dc035640 con 0x7f02dc0c9670 2024-03-29T11:15:16.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.453+0000 7f02ea2e5640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:16.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.453+0000 7f02ea2e5640 10 monclient(hunting): authenticate will time out at 1097.765386s 2024-03-29T11:15:16.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.455+0000 7f02e8ae2640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f02dc035e00 0x7f02dc0c8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:34826/0 (socket says 172.21.15.170:34826) 2024-03-29T11:15:16.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.455+0000 7f02e8ae2640 1 -- 172.21.15.170:0/3864234315 learned_addr learned my addr 172.21.15.170:0/3864234315 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:15:16.434 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.456+0000 7f02dbfff640 1 -- 172.21.15.170:0/3864234315 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356116385 0 0) 0x7f02dc035970 con 0x7f02dc035e00 2024-03-29T11:15:16.434 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.456+0000 7f02dbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:16.434 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.456+0000 7f02dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:16.434 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.456+0000 7f02dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:16.434 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.456+0000 7f02dbfff640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:15:16.434 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.456+0000 7f02dbfff640 1 -- 172.21.15.170:0/3864234315 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d0003520 con 0x7f02dc035e00 2024-03-29T11:15:16.434 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.456+0000 7f02dbfff640 1 -- 172.21.15.170:0/3864234315 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4257264329 0 0) 0x7f02d0003520 con 0x7f02dc035e00 2024-03-29T11:15:16.434 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.456+0000 7f02dbfff640 1 -- 172.21.15.170:0/3864234315 >> v1:172.21.15.170:6790/0 conn(0x7f02dc0c9670 legacy=0x7f02dc0d7560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:16.435 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.456+0000 7f02dbfff640 1 -- 172.21.15.170:0/3864234315 >> v1:172.21.15.184:6789/0 conn(0x7f02dc006e30 legacy=0x7f02dc0cbcc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:16.437 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.458+0000 7f02dbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:16.437 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.458+0000 7f02dbfff640 1 monclient: found mon.noname-a 2024-03-29T11:15:16.437 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.458+0000 7f02dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:16.437 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.458+0000 7f02dbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:15:16.437 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.458+0000 7f02dbfff640 1 -- 172.21.15.170:0/3864234315 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02dc034e20 con 0x7f02dc035e00 2024-03-29T11:15:16.437 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.458+0000 7f02dbfff640 10 monclient: _finish_auth 0 2024-03-29T11:15:16.437 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.458+0000 7f02dbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:16.437 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.458+0000 7f02dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:16.437 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.458+0000 7f02dbfff640 1 -- 172.21.15.170:0/3864234315 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f02e404f730 con 0x7f02dc035e00 2024-03-29T11:15:16.437 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.458+0000 7f02dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:16.437 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.458+0000 7f02dbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:16.437 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.458+0000 7f02dbfff640 10 monclient: dump: 2024-03-29T11:15:16.437 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:16.438 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:16.438 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:16.438 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:16.438 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:16.438 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:16.438 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:16.438 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:16.438 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:16.438 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:16.438 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.459+0000 7f02ea2e5640 5 monclient: authenticate success, global_id 4344 2024-03-29T11:15:16.438 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.459+0000 7f02ea2e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:16.439 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.460+0000 7f02dbfff640 1 -- 172.21.15.170:0/3864234315 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02e40008d0 con 0x7f02dc035e00 2024-03-29T11:15:16.439 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.460+0000 7f02dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:16.439 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.460+0000 7f02dbfff640 1 -- 172.21.15.170:0/3864234315 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f02e40758b0 con 0x7f02dc035e00 2024-03-29T11:15:16.439 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.460+0000 7f02dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:16.439 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.460+0000 7f02dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:16.439 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.460+0000 7f02dbfff640 10 monclient: dump: 2024-03-29T11:15:16.439 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:16.439 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:16.439 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:16.439 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:16.439 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:16.440 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:16.440 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:16.440 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:16.440 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:16.440 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:16.440 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.461+0000 7f02ea2e5640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:16.440 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.461+0000 7f02ea2e5640 4 set_mon_vals no callback set 2024-03-29T11:15:16.440 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.462+0000 7f02ea2e5640 10 monclient: shutdown 2024-03-29T11:15:16.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.462+0000 7f02ea2e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:16.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.462+0000 7f02ea2e5640 1 -- 172.21.15.170:0/3864234315 >> v1:172.21.15.170:6789/0 conn(0x7f02dc035e00 legacy=0x7f02dc0c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:16.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.462+0000 7f02ea2e5640 1 -- 172.21.15.170:0/3864234315 shutdown_connections 2024-03-29T11:15:16.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.462+0000 7f02ea2e5640 1 -- 172.21.15.170:0/3864234315 >> 172.21.15.170:0/3864234315 conn(0x7f02dc005c00 msgr2=0x7f02dc006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:16.446 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.468+0000 7f02ea2e5640 1 -- 172.21.15.170:0/3864234315 shutdown_connections 2024-03-29T11:15:16.446 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.468+0000 7f02ea2e5640 1 -- 172.21.15.170:0/3864234315 wait complete. 2024-03-29T11:15:16.449 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.469+0000 7f02ea2e5640 10 monclient: build_initial_monmap 2024-03-29T11:15:16.449 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.469+0000 7f02ea2e5640 10 monclient: monmap: 2024-03-29T11:15:16.450 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:15:16.450 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:16.450 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:15:16.450 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:15:16.450 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:16.450 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:16.450 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:15:16.450 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:15:16.450 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:15:16.450 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:16.450 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.469+0000 7f02ea2e5640 1 librados: starting msgr at 2024-03-29T11:15:16.450 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.469+0000 7f02ea2e5640 1 librados: starting objecter 2024-03-29T11:15:16.450 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.469+0000 7f02ea2e5640 1 Processor -- start 2024-03-29T11:15:16.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.470+0000 7f02ea2e5640 1 -- start start 2024-03-29T11:15:16.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.470+0000 7f02ea2e5640 1 librados: setting wanted keys 2024-03-29T11:15:16.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.470+0000 7f02ea2e5640 1 librados: calling monclient init 2024-03-29T11:15:16.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.470+0000 7f02ea2e5640 10 monclient: init 2024-03-29T11:15:16.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.470+0000 7f02ea2e5640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:16.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.470+0000 7f02ea2e5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:15:16.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.470+0000 7f02ea2e5640 10 monclient(hunting): picked mon.noname-c con 0x7f02dc006e30 addr v1:172.21.15.170:6790/0 2024-03-29T11:15:16.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.470+0000 7f02ea2e5640 10 monclient(hunting): picked mon.noname-a con 0x7f02dc035e00 addr v1:172.21.15.170:6789/0 2024-03-29T11:15:16.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.470+0000 7f02ea2e5640 10 monclient(hunting): picked mon.noname-b con 0x7f02dc0c9670 addr v1:172.21.15.184:6789/0 2024-03-29T11:15:16.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.470+0000 7f02ea2e5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02dc1699d0 con 0x7f02dc035e00 2024-03-29T11:15:16.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.470+0000 7f02ea2e5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02dc030150 con 0x7f02dc0c9670 2024-03-29T11:15:16.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.470+0000 7f02ea2e5640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02dc169660 con 0x7f02dc006e30 2024-03-29T11:15:16.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.470+0000 7f02ea2e5640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:16.452 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.470+0000 7f02ea2e5640 10 monclient(hunting): authenticate will time out at 1097.782865s 2024-03-29T11:15:16.452 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.475+0000 7f02e8ae2640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f02dc035e00 0x7f02dc160420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:36756/0 (socket says 172.21.15.170:36756) 2024-03-29T11:15:16.452 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.475+0000 7f02e8ae2640 1 -- 172.21.15.170:0/3679930709 learned_addr learned my addr 172.21.15.170:0/3679930709 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:15:16.453 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.475+0000 7f02d9ffb640 10 client.?.objecter ms_handle_connect 0x7f02dc035e00 2024-03-29T11:15:16.454 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.475+0000 7f02d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:16.454 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.475+0000 7f02d9ffb640 1 -- 172.21.15.170:0/3679930709 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751231374 0 0) 0x7f02dc1699d0 con 0x7f02dc035e00 2024-03-29T11:15:16.454 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.475+0000 7f02d9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:16.454 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.475+0000 7f02d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:16.454 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.475+0000 7f02d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:16.454 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.475+0000 7f02d9ffb640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:15:16.454 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.475+0000 7f02d9ffb640 1 -- 172.21.15.170:0/3679930709 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c8004eb0 con 0x7f02dc035e00 2024-03-29T11:15:16.455 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.476+0000 7f02d9ffb640 10 client.?.objecter ms_handle_connect 0x7f02dc006e30 2024-03-29T11:15:16.455 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.476+0000 7f02d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:16.456 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.476+0000 7f02d9ffb640 1 -- 172.21.15.170:0/3679930709 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131416618 0 0) 0x7f02dc169660 con 0x7f02dc006e30 2024-03-29T11:15:16.456 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.476+0000 7f02d9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:16.456 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.476+0000 7f02d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:16.456 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.476+0000 7f02d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:16.456 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.476+0000 7f02d9ffb640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:15:16.456 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.476+0000 7f02d9ffb640 1 -- 172.21.15.170:0/3679930709 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02dc1699d0 con 0x7f02dc006e30 2024-03-29T11:15:16.456 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.476+0000 7f02d9ffb640 1 -- 172.21.15.170:0/3679930709 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1315189880 0 0) 0x7f02c8004eb0 con 0x7f02dc035e00 2024-03-29T11:15:16.456 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.476+0000 7f02d9ffb640 1 -- 172.21.15.170:0/3679930709 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02dc169660 con 0x7f02dc035e00 2024-03-29T11:15:16.456 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.476+0000 7f02d9ffb640 1 -- 172.21.15.170:0/3679930709 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f02e40758b0 con 0x7f02dc035e00 2024-03-29T11:15:16.456 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.476+0000 7f02d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:16.456 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.476+0000 7f02d9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:16.456 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.476+0000 7f02d9ffb640 10 monclient(hunting): dump: 2024-03-29T11:15:16.456 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:16.456 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:16.456 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:16.457 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:16.457 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:16.457 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:16.457 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:16.457 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:16.457 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:16.457 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:16.457 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.476+0000 7f02d9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:16.457 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.476+0000 7f02d9ffb640 10 client.?.objecter ms_handle_connect 0x7f02dc0c9670 2024-03-29T11:15:16.457 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.476+0000 7f02d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:16.457 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02d9ffb640 1 -- 172.21.15.170:0/3679930709 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154112851 0 0) 0x7f02dc030150 con 0x7f02dc0c9670 2024-03-29T11:15:16.457 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02d9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:16.457 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:16.457 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:16.457 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02d9ffb640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:15:16.458 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02d9ffb640 1 -- 172.21.15.170:0/3679930709 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c8004eb0 con 0x7f02dc0c9670 2024-03-29T11:15:16.458 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02d9ffb640 1 -- 172.21.15.170:0/3679930709 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 561926153 0 0) 0x7f02dc169660 con 0x7f02dc035e00 2024-03-29T11:15:16.458 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02d9ffb640 1 -- 172.21.15.170:0/3679930709 >> v1:172.21.15.170:6790/0 conn(0x7f02dc006e30 legacy=0x7f02dc169ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:16.458 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02d9ffb640 1 -- 172.21.15.170:0/3679930709 >> v1:172.21.15.184:6789/0 conn(0x7f02dc0c9670 legacy=0x7f02dc174ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:16.458 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02d9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:15:16.458 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02d9ffb640 1 monclient: found mon.a 2024-03-29T11:15:16.458 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:16.458 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02d9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:15:16.458 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02d9ffb640 1 -- 172.21.15.170:0/3679930709 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02dc032a90 con 0x7f02dc035e00 2024-03-29T11:15:16.458 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02d9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:15:16.458 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02d9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:15:16.458 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:16.458 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02ea2e5640 5 monclient: authenticate success, global_id 4347 2024-03-29T11:15:16.458 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02ea2e5640 10 monclient: _renew_subs 2024-03-29T11:15:16.459 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02ea2e5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:15:16.459 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.477+0000 7f02ea2e5640 1 -- 172.21.15.170:0/3679930709 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02dc032d40 con 0x7f02dc035e00 2024-03-29T11:15:16.461 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.478+0000 7f02ea2e5640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:16.461 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.478+0000 7f02ea2e5640 10 monclient: _renew_subs 2024-03-29T11:15:16.461 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.478+0000 7f02ea2e5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:15:16.461 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.478+0000 7f02ea2e5640 1 -- 172.21.15.170:0/3679930709 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02dc02edc0 con 0x7f02dc035e00 2024-03-29T11:15:16.461 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.478+0000 7f02ea2e5640 1 librados: init done 2024-03-29T11:15:16.461 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.479+0000 7f02ea2e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:16.461 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.479+0000 7f02ea2e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:16.461 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.479+0000 7f02ea2e5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:15:16.462 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.479+0000 7f02ea2e5640 1 -- 172.21.15.170:0/3679930709 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02dc007b90 con 0x7f02dc035e00 2024-03-29T11:15:16.462 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.480+0000 7f02d9ffb640 1 -- 172.21.15.170:0/3679930709 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02e404f400 con 0x7f02dc035e00 2024-03-29T11:15:16.462 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.480+0000 7f02d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:16.462 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.480+0000 7f02d9ffb640 1 -- 172.21.15.170:0/3679930709 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f02e404fdd0 con 0x7f02dc035e00 2024-03-29T11:15:16.462 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.480+0000 7f02daffd640 4 set_mon_vals no callback set 2024-03-29T11:15:16.462 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.481+0000 7f02d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:16.462 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.481+0000 7f02d9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:16.462 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.481+0000 7f02d9ffb640 10 monclient: dump: 2024-03-29T11:15:16.462 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:16.462 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:16.462 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:16.462 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:16.462 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:16.463 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:16.463 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:16.463 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:16.463 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:16.463 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:16.463 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.481+0000 7f02d9ffb640 1 -- 172.21.15.170:0/3679930709 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f02e4067ea0 con 0x7f02dc035e00 2024-03-29T11:15:16.463 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.482+0000 7f02e92e3640 10 monclient: get_auth_request con 0x7f02c806da80 auth_method 0 2024-03-29T11:15:16.463 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.482+0000 7f02d9ffb640 1 -- 172.21.15.170:0/3679930709 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f02e41006f0 con 0x7f02dc035e00 2024-03-29T11:15:16.463 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.482+0000 7f02d9ffb640 10 client.4347.objecter ms_dispatch 0x7f02e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:16.463 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.482+0000 7f02d9ffb640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:16.463 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.482+0000 7f02d9ffb640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:16.463 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.483+0000 7f02d9ffb640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-29T11:15:16.463 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.483+0000 7f02d9ffb640 10 client.4347.objecter ms_handle_connect 0x7f02c806da80 2024-03-29T11:15:16.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.491+0000 7f02d9ffb640 1 -- 172.21.15.170:0/3679930709 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f02e40cb020 con 0x7f02dc035e00 2024-03-29T11:15:16.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.491+0000 7f02d9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:16.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.491+0000 7f02d9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:16.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.814+0000 7f02ea2e5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:16.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.814+0000 7f02ea2e5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:16.795 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.814+0000 7f02ea2e5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:15:16.795 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.814+0000 7f02ea2e5640 1 -- 172.21.15.170:0/3679930709 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02dc169b60 con 0x7f02dc035e00 2024-03-29T11:15:16.795 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.818+0000 7f02d9ffb640 1 -- 172.21.15.170:0/3679930709 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f02e40ccf50 con 0x7f02dc035e00 2024-03-29T11:15:16.796 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.818+0000 7f02d9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:16.796 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.818+0000 7f02d9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:16.797 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:15:16.797 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:16.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.821+0000 7f02ea2e5640 10 librados: watch_flush enter 2024-03-29T11:15:16.801 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.823+0000 7f02ea2e5640 10 librados: watch_flush exit 2024-03-29T11:15:16.801 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.823+0000 7f02ea2e5640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-29T11:15:16.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.823+0000 7f02ea2e5640 10 client.4347.objecter successfully canceled tick 2024-03-29T11:15:16.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.823+0000 7f02ea2e5640 1 -- 172.21.15.170:0/3679930709 >> v1:172.21.15.170:6816/3298994444 conn(0x7f02c806da80 legacy=0x7f02c806ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:16.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.823+0000 7f02ea2e5640 10 monclient: shutdown 2024-03-29T11:15:16.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.823+0000 7f02ea2e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:16.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.823+0000 7f02ea2e5640 1 -- 172.21.15.170:0/3679930709 >> v1:172.21.15.170:6789/0 conn(0x7f02dc035e00 legacy=0x7f02dc160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:16.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.824+0000 7f02ea2e5640 1 -- 172.21.15.170:0/3679930709 shutdown_connections 2024-03-29T11:15:16.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.824+0000 7f02ea2e5640 1 -- 172.21.15.170:0/3679930709 >> 172.21.15.170:0/3679930709 conn(0x7f02dc005c00 msgr2=0x7f02dc0c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:16.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.824+0000 7f02ea2e5640 1 -- 172.21.15.170:0/3679930709 shutdown_connections 2024-03-29T11:15:16.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.824+0000 7f02ea2e5640 1 -- 172.21.15.170:0/3679930709 wait complete. 2024-03-29T11:15:16.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:16.824+0000 7f02ea2e5640 1 librados: shutdown 2024-03-29T11:15:16.820 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:23.627 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:23.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.813+0000 7fb20ad2b640 10 monclient: get_monmap_and_config 2024-03-29T11:15:23.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.813+0000 7fb20ad2b640 10 monclient: build_initial_monmap 2024-03-29T11:15:23.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.813+0000 7fb20ad2b640 10 monclient: monmap: 2024-03-29T11:15:23.792 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:15:23.792 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:23.792 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:15:23.815322+0000 2024-03-29T11:15:23.792 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:15:23.815322+0000 2024-03-29T11:15:23.792 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:23.792 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:23.792 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:15:23.792 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:15:23.792 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:15:23.793 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:23.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.815+0000 7fb20ad2b640 1 Processor -- start 2024-03-29T11:15:23.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.815+0000 7fb20ad2b640 1 -- start start 2024-03-29T11:15:23.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.815+0000 7fb20ad2b640 10 monclient: init 2024-03-29T11:15:23.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.815+0000 7fb20ad2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:23.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.815+0000 7fb20ad2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:15:23.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.815+0000 7fb20ad2b640 10 monclient(hunting): picked mon.noname-b con 0x7fb2041457f0 addr v1:172.21.15.170:6790/0 2024-03-29T11:15:23.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.815+0000 7fb20ad2b640 10 monclient(hunting): picked mon.noname-c con 0x7fb20413f650 addr v1:172.21.15.184:6789/0 2024-03-29T11:15:23.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.815+0000 7fb20ad2b640 10 monclient(hunting): picked mon.noname-a con 0x7fb2040a9d70 addr v1:172.21.15.170:6789/0 2024-03-29T11:15:23.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.815+0000 7fb20ad2b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2040aaef0 con 0x7fb2040a9d70 2024-03-29T11:15:23.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.815+0000 7fb20ad2b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb204145bd0 con 0x7fb20413f650 2024-03-29T11:15:23.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.815+0000 7fb20ad2b640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb204145eb0 con 0x7fb2041457f0 2024-03-29T11:15:23.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.815+0000 7fb20ad2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:23.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.815+0000 7fb20ad2b640 10 monclient(hunting): authenticate will time out at 1105.127932s 2024-03-29T11:15:23.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.815+0000 7fb208ac7640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fb2041457f0 0x7fb20413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:58492/0 (socket says 172.21.15.170:58492) 2024-03-29T11:15:23.795 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.815+0000 7fb208ac7640 1 -- 172.21.15.170:0/857772490 learned_addr learned my addr 172.21.15.170:0/857772490 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:15:23.796 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.818+0000 7fb2037fe640 1 -- 172.21.15.170:0/857772490 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490375275 0 0) 0x7fb204145bd0 con 0x7fb20413f650 2024-03-29T11:15:23.796 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.818+0000 7fb2037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:23.796 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.818+0000 7fb2037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:23.796 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.818+0000 7fb2037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:23.797 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.818+0000 7fb2037fe640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:15:23.797 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.818+0000 7fb2037fe640 1 -- 172.21.15.170:0/857772490 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1ec003520 con 0x7fb20413f650 2024-03-29T11:15:23.798 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.819+0000 7fb2037fe640 1 -- 172.21.15.170:0/857772490 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254783513 0 0) 0x7fb2040aaef0 con 0x7fb2040a9d70 2024-03-29T11:15:23.798 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.819+0000 7fb2037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:23.798 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.819+0000 7fb2037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:23.798 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.819+0000 7fb2037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:23.798 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.819+0000 7fb2037fe640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:15:23.798 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.819+0000 7fb2037fe640 1 -- 172.21.15.170:0/857772490 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb204145bd0 con 0x7fb2040a9d70 2024-03-29T11:15:23.798 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.819+0000 7fb2037fe640 1 -- 172.21.15.170:0/857772490 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2289076496 0 0) 0x7fb204145eb0 con 0x7fb2041457f0 2024-03-29T11:15:23.798 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:23.798 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:23.798 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:23.798 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:15:23.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 1 -- 172.21.15.170:0/857772490 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2040aaef0 con 0x7fb2041457f0 2024-03-29T11:15:23.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 1 -- 172.21.15.170:0/857772490 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3785156548 0 0) 0x7fb1ec003520 con 0x7fb20413f650 2024-03-29T11:15:23.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 1 -- 172.21.15.170:0/857772490 >> v1:172.21.15.170:6790/0 conn(0x7fb2041457f0 legacy=0x7fb20413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:23.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 1 -- 172.21.15.170:0/857772490 >> v1:172.21.15.170:6789/0 conn(0x7fb2040a9d70 legacy=0x7fb20414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:23.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 10 monclient: _finish_hunting 0 2024-03-29T11:15:23.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 1 monclient: found mon.noname-c 2024-03-29T11:15:23.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:23.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:15:23.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 1 -- 172.21.15.170:0/857772490 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb204146340 con 0x7fb20413f650 2024-03-29T11:15:23.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 10 monclient: _finish_auth 0 2024-03-29T11:15:23.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 10 monclient: _check_auth_tickets 2024-03-29T11:15:23.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:23.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 1 -- 172.21.15.170:0/857772490 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb1f4001610 con 0x7fb20413f650 2024-03-29T11:15:23.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:23.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:23.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 10 monclient: dump: 2024-03-29T11:15:23.800 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:23.800 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:23.800 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:23.800 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:23.800 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:23.800 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:23.800 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:23.800 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:23.800 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:23.800 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:23.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb20ad2b640 5 monclient: authenticate success, global_id 4348 2024-03-29T11:15:23.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb20ad2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:23.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 1 -- 172.21.15.170:0/857772490 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1f4001770 con 0x7fb20413f650 2024-03-29T11:15:23.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:23.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 1 -- 172.21.15.170:0/857772490 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb1f4001bc0 con 0x7fb20413f650 2024-03-29T11:15:23.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb20ad2b640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:23.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb20ad2b640 4 set_mon_vals no callback set 2024-03-29T11:15:23.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:23.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.820+0000 7fb2037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:23.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.821+0000 7fb2037fe640 10 monclient: dump: 2024-03-29T11:15:23.803 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:23.803 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:23.803 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:23.803 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:23.803 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:23.803 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:23.803 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:23.803 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:23.803 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:23.803 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:23.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.821+0000 7fb20ad2b640 10 monclient: shutdown 2024-03-29T11:15:23.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.821+0000 7fb20ad2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:23.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.821+0000 7fb20ad2b640 1 -- 172.21.15.170:0/857772490 >> v1:172.21.15.184:6789/0 conn(0x7fb20413f650 legacy=0x7fb2040b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:23.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.822+0000 7fb20ad2b640 1 -- 172.21.15.170:0/857772490 shutdown_connections 2024-03-29T11:15:23.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.822+0000 7fb20ad2b640 1 -- 172.21.15.170:0/857772490 >> 172.21.15.170:0/857772490 conn(0x7fb204066a00 msgr2=0x7fb204066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:23.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.822+0000 7fb20ad2b640 1 -- 172.21.15.170:0/857772490 shutdown_connections 2024-03-29T11:15:23.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.823+0000 7fb20ad2b640 1 -- 172.21.15.170:0/857772490 wait complete. 2024-03-29T11:15:23.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.824+0000 7fb20ad2b640 10 monclient: build_initial_monmap 2024-03-29T11:15:23.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.824+0000 7fb20ad2b640 10 monclient: monmap: 2024-03-29T11:15:23.804 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:15:23.804 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:23.804 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:15:23.804 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:15:23.804 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:23.804 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:23.804 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:15:23.805 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:15:23.805 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:15:23.805 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:23.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.824+0000 7fb20ad2b640 1 librados: starting msgr at 2024-03-29T11:15:23.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.824+0000 7fb20ad2b640 1 librados: starting objecter 2024-03-29T11:15:23.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.824+0000 7fb20ad2b640 1 Processor -- start 2024-03-29T11:15:23.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.825+0000 7fb20ad2b640 1 -- start start 2024-03-29T11:15:23.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.825+0000 7fb20ad2b640 1 librados: setting wanted keys 2024-03-29T11:15:23.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.825+0000 7fb20ad2b640 1 librados: calling monclient init 2024-03-29T11:15:23.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.825+0000 7fb20ad2b640 10 monclient: init 2024-03-29T11:15:23.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.826+0000 7fb20ad2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:23.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.826+0000 7fb20ad2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:15:23.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.826+0000 7fb20ad2b640 10 monclient(hunting): picked mon.noname-c con 0x7fb2040a9d70 addr v1:172.21.15.170:6790/0 2024-03-29T11:15:23.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.826+0000 7fb20ad2b640 10 monclient(hunting): picked mon.noname-b con 0x7fb2041457f0 addr v1:172.21.15.184:6789/0 2024-03-29T11:15:23.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.826+0000 7fb20ad2b640 10 monclient(hunting): picked mon.noname-a con 0x7fb204139580 addr v1:172.21.15.170:6789/0 2024-03-29T11:15:23.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.826+0000 7fb20ad2b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb20413b010 con 0x7fb204139580 2024-03-29T11:15:23.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.826+0000 7fb20ad2b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb204142c80 con 0x7fb2041457f0 2024-03-29T11:15:23.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.826+0000 7fb20ad2b640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb20413aca0 con 0x7fb2040a9d70 2024-03-29T11:15:23.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.826+0000 7fb20ad2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:23.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.826+0000 7fb20ad2b640 10 monclient(hunting): authenticate will time out at 1105.138293s 2024-03-29T11:15:23.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.826+0000 7fb208ac7640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fb2040a9d70 0x7fb204138e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:58728/0 (socket says 172.21.15.170:58728) 2024-03-29T11:15:23.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.826+0000 7fb208ac7640 1 -- 172.21.15.170:0/1929426669 learned_addr learned my addr 172.21.15.170:0/1929426669 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:15:23.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.826+0000 7fb2092c8640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fb204139580 0x7fb2041efb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:41702/0 (socket says 172.21.15.170:41702) 2024-03-29T11:15:23.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.827+0000 7fb2017fa640 10 client.?.objecter ms_handle_connect 0x7fb204139580 2024-03-29T11:15:23.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.827+0000 7fb2017fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:23.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.827+0000 7fb2017fa640 10 client.?.objecter ms_handle_connect 0x7fb2040a9d70 2024-03-29T11:15:23.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.827+0000 7fb2017fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:23.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.827+0000 7fb2017fa640 10 client.?.objecter ms_handle_connect 0x7fb2041457f0 2024-03-29T11:15:23.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.827+0000 7fb2017fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:23.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.827+0000 7fb2017fa640 1 -- 172.21.15.170:0/1929426669 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628354941 0 0) 0x7fb204142c80 con 0x7fb2041457f0 2024-03-29T11:15:23.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.827+0000 7fb2017fa640 10 monclient(hunting): _negotiate 2024-03-29T11:15:23.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.827+0000 7fb2017fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:23.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.827+0000 7fb2017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:23.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.827+0000 7fb2017fa640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:15:23.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.827+0000 7fb2017fa640 1 -- 172.21.15.170:0/1929426669 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f0004eb0 con 0x7fb2041457f0 2024-03-29T11:15:23.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.827+0000 7fb2017fa640 1 -- 172.21.15.170:0/1929426669 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422543292 0 0) 0x7fb20413aca0 con 0x7fb2040a9d70 2024-03-29T11:15:23.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 10 monclient(hunting): _negotiate 2024-03-29T11:15:23.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:23.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:23.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:15:23.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 1 -- 172.21.15.170:0/1929426669 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb204142c80 con 0x7fb2040a9d70 2024-03-29T11:15:23.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 1 -- 172.21.15.170:0/1929426669 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268962674 0 0) 0x7fb20413b010 con 0x7fb204139580 2024-03-29T11:15:23.815 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 10 monclient(hunting): _negotiate 2024-03-29T11:15:23.815 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:23.815 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:23.815 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:15:23.815 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 1 -- 172.21.15.170:0/1929426669 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb20413aca0 con 0x7fb204139580 2024-03-29T11:15:23.815 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 1 -- 172.21.15.170:0/1929426669 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1570391921 0 0) 0x7fb1f0004eb0 con 0x7fb2041457f0 2024-03-29T11:15:23.815 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 1 -- 172.21.15.170:0/1929426669 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb20413b010 con 0x7fb2041457f0 2024-03-29T11:15:23.815 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 1 -- 172.21.15.170:0/1929426669 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb1f4001a80 con 0x7fb2041457f0 2024-03-29T11:15:23.815 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:23.815 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:23.815 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 10 monclient(hunting): dump: 2024-03-29T11:15:23.815 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:23.815 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:23.815 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:23.815 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:23.816 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:23.816 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:23.816 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:23.816 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:23.816 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:23.816 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:23.816 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:23.816 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 1 -- 172.21.15.170:0/1929426669 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 897558357 0 0) 0x7fb20413b010 con 0x7fb2041457f0 2024-03-29T11:15:23.816 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 1 -- 172.21.15.170:0/1929426669 >> v1:172.21.15.170:6790/0 conn(0x7fb2040a9d70 legacy=0x7fb204138e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:23.816 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 1 -- 172.21.15.170:0/1929426669 >> v1:172.21.15.170:6789/0 conn(0x7fb204139580 legacy=0x7fb2041efb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:23.816 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 10 monclient: _finish_hunting 0 2024-03-29T11:15:23.816 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 1 monclient: found mon.b 2024-03-29T11:15:23.816 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.828+0000 7fb2017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:23.816 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb2017fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:15:23.816 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb2017fa640 1 -- 172.21.15.170:0/1929426669 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20413b530 con 0x7fb2041457f0 2024-03-29T11:15:23.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb2017fa640 10 monclient: _finish_auth 0 2024-03-29T11:15:23.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb2017fa640 10 monclient: _check_auth_tickets 2024-03-29T11:15:23.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb2017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:23.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb20ad2b640 5 monclient: authenticate success, global_id 4351 2024-03-29T11:15:23.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb20ad2b640 10 monclient: _renew_subs 2024-03-29T11:15:23.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb20ad2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:15:23.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb20ad2b640 1 -- 172.21.15.170:0/1929426669 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20413a200 con 0x7fb2041457f0 2024-03-29T11:15:23.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb20ad2b640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:23.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb20ad2b640 10 monclient: _renew_subs 2024-03-29T11:15:23.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb20ad2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:15:23.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb20ad2b640 1 -- 172.21.15.170:0/1929426669 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb204141a40 con 0x7fb2041457f0 2024-03-29T11:15:23.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb20ad2b640 1 librados: init done 2024-03-29T11:15:23.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb2017fa640 1 -- 172.21.15.170:0/1929426669 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1f4001c00 con 0x7fb2041457f0 2024-03-29T11:15:23.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb2017fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:23.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb2017fa640 1 -- 172.21.15.170:0/1929426669 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb1f4001250 con 0x7fb2041457f0 2024-03-29T11:15:23.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb2017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:23.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb2017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:23.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb2017fa640 10 monclient: dump: 2024-03-29T11:15:23.818 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:23.818 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:23.818 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:23.818 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:23.818 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:23.818 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:23.818 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:23.818 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:23.818 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:23.818 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:23.819 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.829+0000 7fb2027fc640 4 set_mon_vals no callback set 2024-03-29T11:15:23.819 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.831+0000 7fb2017fa640 1 -- 172.21.15.170:0/1929426669 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fb1f40198b0 con 0x7fb2041457f0 2024-03-29T11:15:23.819 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.832+0000 7fb20ad2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:23.819 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.832+0000 7fb20ad2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:23.819 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.832+0000 7fb20ad2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:15:23.819 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.832+0000 7fb20ad2b640 1 -- 172.21.15.170:0/1929426669 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e4005080 con 0x7fb2041457f0 2024-03-29T11:15:23.819 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.832+0000 7fb208ac7640 10 monclient: get_auth_request con 0x7fb1f006dea0 auth_method 0 2024-03-29T11:15:23.819 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.832+0000 7fb2017fa640 1 -- 172.21.15.170:0/1929426669 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fb1f40837e0 con 0x7fb2041457f0 2024-03-29T11:15:23.819 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.832+0000 7fb2017fa640 10 client.4351.objecter ms_dispatch 0x7fb204000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:23.819 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.832+0000 7fb2017fa640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:23.819 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.832+0000 7fb2017fa640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:23.819 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.832+0000 7fb2017fa640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-29T11:15:23.819 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.833+0000 7fb2017fa640 10 client.4351.objecter ms_handle_connect 0x7fb1f006dea0 2024-03-29T11:15:23.820 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.841+0000 7fb2017fa640 1 -- 172.21.15.170:0/1929426669 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1f404eef0 con 0x7fb2041457f0 2024-03-29T11:15:23.820 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.841+0000 7fb2017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:23.820 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:23.841+0000 7fb2017fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:24.244 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.266+0000 7fb20ad2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:24.244 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.266+0000 7fb20ad2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:24.245 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.266+0000 7fb20ad2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:15:24.245 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.266+0000 7fb20ad2b640 1 -- 172.21.15.170:0/1929426669 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1e4005330 con 0x7fb2041457f0 2024-03-29T11:15:24.245 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.267+0000 7fb2017fa640 1 -- 172.21.15.170:0/1929426669 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fb1f4050ff0 con 0x7fb2041457f0 2024-03-29T11:15:24.245 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.267+0000 7fb2017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:24.245 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.267+0000 7fb2017fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:24.245 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:15:24.246 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:24.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.271+0000 7fb20ad2b640 10 librados: watch_flush enter 2024-03-29T11:15:24.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.272+0000 7fb20ad2b640 10 librados: watch_flush exit 2024-03-29T11:15:24.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.272+0000 7fb20ad2b640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-29T11:15:24.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.272+0000 7fb20ad2b640 10 client.4351.objecter successfully canceled tick 2024-03-29T11:15:24.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.272+0000 7fb20ad2b640 1 -- 172.21.15.170:0/1929426669 >> v1:172.21.15.170:6816/3298994444 conn(0x7fb1f006dea0 legacy=0x7fb1f0070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:24.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.272+0000 7fb20ad2b640 10 monclient: shutdown 2024-03-29T11:15:24.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.272+0000 7fb20ad2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:24.252 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.272+0000 7fb20ad2b640 1 -- 172.21.15.170:0/1929426669 >> v1:172.21.15.184:6789/0 conn(0x7fb2041457f0 legacy=0x7fb204134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:24.252 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.272+0000 7fb20ad2b640 1 -- 172.21.15.170:0/1929426669 shutdown_connections 2024-03-29T11:15:24.252 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.272+0000 7fb20ad2b640 1 -- 172.21.15.170:0/1929426669 >> 172.21.15.170:0/1929426669 conn(0x7fb204066a00 msgr2=0x7fb2040b4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:24.252 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.272+0000 7fb20ad2b640 1 -- 172.21.15.170:0/1929426669 shutdown_connections 2024-03-29T11:15:24.252 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.272+0000 7fb20ad2b640 1 -- 172.21.15.170:0/1929426669 wait complete. 2024-03-29T11:15:24.252 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:24.272+0000 7fb20ad2b640 1 librados: shutdown 2024-03-29T11:15:24.267 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:31.076 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:31.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.243+0000 7f8259a63640 10 monclient: get_monmap_and_config 2024-03-29T11:15:31.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.243+0000 7f8259a63640 10 monclient: build_initial_monmap 2024-03-29T11:15:31.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.243+0000 7f8259a63640 10 monclient: monmap: 2024-03-29T11:15:31.221 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:15:31.221 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:31.221 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:15:31.243715+0000 2024-03-29T11:15:31.221 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:15:31.243715+0000 2024-03-29T11:15:31.221 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:31.221 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:31.221 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:15:31.221 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:15:31.221 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:15:31.221 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:31.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.244+0000 7f8259a63640 1 Processor -- start 2024-03-29T11:15:31.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.244+0000 7f8259a63640 1 -- start start 2024-03-29T11:15:31.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.244+0000 7f8259a63640 10 monclient: init 2024-03-29T11:15:31.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.245+0000 7f8259a63640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:31.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.245+0000 7f8259a63640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:15:31.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.245+0000 7f8259a63640 10 monclient(hunting): picked mon.noname-a con 0x7f82540b4330 addr v1:172.21.15.170:6789/0 2024-03-29T11:15:31.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.245+0000 7f8259a63640 10 monclient(hunting): picked mon.noname-b con 0x7f82541470b0 addr v1:172.21.15.170:6790/0 2024-03-29T11:15:31.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.245+0000 7f8259a63640 10 monclient(hunting): picked mon.noname-c con 0x7f82541439b0 addr v1:172.21.15.184:6789/0 2024-03-29T11:15:31.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.245+0000 7f8259a63640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825406f2a0 con 0x7f82540b4330 2024-03-29T11:15:31.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.245+0000 7f8259a63640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8254146b90 con 0x7f82541439b0 2024-03-29T11:15:31.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.245+0000 7f8259a63640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82540b2ca0 con 0x7f82541470b0 2024-03-29T11:15:31.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.245+0000 7f8259a63640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:31.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.245+0000 7f8259a63640 10 monclient(hunting): authenticate will time out at 1112.556585s 2024-03-29T11:15:31.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.246+0000 7f8252ffd640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f82541470b0 0x7f82540b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:35882/0 (socket says 172.21.15.170:35882) 2024-03-29T11:15:31.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.246+0000 7f8252ffd640 1 -- 172.21.15.170:0/2065823326 learned_addr learned my addr 172.21.15.170:0/2065823326 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:15:31.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.246+0000 7f82527fc640 1 -- 172.21.15.170:0/2065823326 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279007376 0 0) 0x7f8254146b90 con 0x7f82541439b0 2024-03-29T11:15:31.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.246+0000 7f82527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:31.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.246+0000 7f82527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:31.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.246+0000 7f82527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:31.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.246+0000 7f82527fc640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:15:31.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.246+0000 7f82527fc640 1 -- 172.21.15.170:0/2065823326 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f823c003520 con 0x7f82541439b0 2024-03-29T11:15:31.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 1 -- 172.21.15.170:0/2065823326 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568936779 0 0) 0x7f82540b2ca0 con 0x7f82541470b0 2024-03-29T11:15:31.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:31.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:31.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:31.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:15:31.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 1 -- 172.21.15.170:0/2065823326 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8254146b90 con 0x7f82541470b0 2024-03-29T11:15:31.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 1 -- 172.21.15.170:0/2065823326 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561818577 0 0) 0x7f825406f2a0 con 0x7f82540b4330 2024-03-29T11:15:31.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:31.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:31.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:31.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:15:31.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 1 -- 172.21.15.170:0/2065823326 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82540b2ca0 con 0x7f82540b4330 2024-03-29T11:15:31.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 1 -- 172.21.15.170:0/2065823326 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4092084633 0 0) 0x7f823c003520 con 0x7f82541439b0 2024-03-29T11:15:31.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 1 -- 172.21.15.170:0/2065823326 >> v1:172.21.15.170:6790/0 conn(0x7f82541470b0 legacy=0x7f82540b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:31.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 1 -- 172.21.15.170:0/2065823326 >> v1:172.21.15.170:6789/0 conn(0x7f82540b4330 legacy=0x7f82540b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:31.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 10 monclient: _finish_hunting 0 2024-03-29T11:15:31.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 1 monclient: found mon.noname-c 2024-03-29T11:15:31.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:31.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:15:31.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 1 -- 172.21.15.170:0/2065823326 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825406e0c0 con 0x7f82541439b0 2024-03-29T11:15:31.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 10 monclient: _finish_auth 0 2024-03-29T11:15:31.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 10 monclient: _check_auth_tickets 2024-03-29T11:15:31.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:31.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 1 -- 172.21.15.170:0/2065823326 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f824c001a10 con 0x7f82541439b0 2024-03-29T11:15:31.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:31.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:31.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f82527fc640 10 monclient: dump: 2024-03-29T11:15:31.228 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:31.229 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:31.229 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:31.229 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:31.229 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:31.229 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:31.229 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:31.229 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:31.229 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:31.229 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:31.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f8259a63640 5 monclient: authenticate success, global_id 4354 2024-03-29T11:15:31.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.247+0000 7f8259a63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:31.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.248+0000 7f82527fc640 1 -- 172.21.15.170:0/2065823326 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f824c001b70 con 0x7f82541439b0 2024-03-29T11:15:31.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.248+0000 7f82527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:31.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.248+0000 7f82527fc640 1 -- 172.21.15.170:0/2065823326 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f824c001f80 con 0x7f82541439b0 2024-03-29T11:15:31.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.248+0000 7f82527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:31.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.248+0000 7f82527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:31.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.248+0000 7f82527fc640 10 monclient: dump: 2024-03-29T11:15:31.230 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:31.230 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:31.230 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:31.230 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:31.231 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:31.231 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:31.231 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:31.231 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:31.231 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:31.231 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:31.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.248+0000 7f8259a63640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:31.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.248+0000 7f8259a63640 4 set_mon_vals no callback set 2024-03-29T11:15:31.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.249+0000 7f8259a63640 10 monclient: shutdown 2024-03-29T11:15:31.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.249+0000 7f8259a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:31.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.249+0000 7f8259a63640 1 -- 172.21.15.170:0/2065823326 >> v1:172.21.15.184:6789/0 conn(0x7f82541439b0 legacy=0x7f8254151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:31.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.249+0000 7f8259a63640 1 -- 172.21.15.170:0/2065823326 shutdown_connections 2024-03-29T11:15:31.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.249+0000 7f8259a63640 1 -- 172.21.15.170:0/2065823326 >> 172.21.15.170:0/2065823326 conn(0x7f8254066a00 msgr2=0x7f8254066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:31.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.249+0000 7f8259a63640 1 -- 172.21.15.170:0/2065823326 shutdown_connections 2024-03-29T11:15:31.240 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.249+0000 7f8259a63640 1 -- 172.21.15.170:0/2065823326 wait complete. 2024-03-29T11:15:31.240 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.250+0000 7f8259a63640 10 monclient: build_initial_monmap 2024-03-29T11:15:31.240 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.250+0000 7f8259a63640 10 monclient: monmap: 2024-03-29T11:15:31.240 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:15:31.240 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:31.240 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:15:31.241 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:15:31.241 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:31.241 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:31.241 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:15:31.241 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:15:31.241 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:15:31.241 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:31.241 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.251+0000 7f8259a63640 1 librados: starting msgr at 2024-03-29T11:15:31.241 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.251+0000 7f8259a63640 1 librados: starting objecter 2024-03-29T11:15:31.241 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.251+0000 7f8259a63640 1 Processor -- start 2024-03-29T11:15:31.241 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.251+0000 7f8259a63640 1 -- start start 2024-03-29T11:15:31.241 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.251+0000 7f8259a63640 1 librados: setting wanted keys 2024-03-29T11:15:31.241 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.251+0000 7f8259a63640 1 librados: calling monclient init 2024-03-29T11:15:31.242 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.251+0000 7f8259a63640 10 monclient: init 2024-03-29T11:15:31.242 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.252+0000 7f8259a63640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:31.242 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.252+0000 7f8259a63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:15:31.242 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.252+0000 7f8259a63640 10 monclient(hunting): picked mon.noname-c con 0x7f82540b4330 addr v1:172.21.15.170:6790/0 2024-03-29T11:15:31.242 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.252+0000 7f8259a63640 10 monclient(hunting): picked mon.noname-b con 0x7f82541439b0 addr v1:172.21.15.184:6789/0 2024-03-29T11:15:31.242 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.252+0000 7f8259a63640 10 monclient(hunting): picked mon.noname-a con 0x7f82541470b0 addr v1:172.21.15.170:6789/0 2024-03-29T11:15:31.242 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.252+0000 7f8259a63640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8254122900 con 0x7f82541470b0 2024-03-29T11:15:31.242 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.252+0000 7f8259a63640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82541274d0 con 0x7f82541439b0 2024-03-29T11:15:31.242 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.252+0000 7f8259a63640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8254126540 con 0x7f82540b4330 2024-03-29T11:15:31.242 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.252+0000 7f8259a63640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:31.242 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.252+0000 7f8259a63640 10 monclient(hunting): authenticate will time out at 1112.563478s 2024-03-29T11:15:31.243 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.252+0000 7f8252ffd640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f82541439b0 0x7f82541db9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:37638/0 (socket says 172.21.15.170:37638) 2024-03-29T11:15:31.243 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.252+0000 7f8252ffd640 1 -- 172.21.15.170:0/658446904 learned_addr learned my addr 172.21.15.170:0/658446904 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:15:31.243 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.252+0000 7f823bfff640 10 client.?.objecter ms_handle_connect 0x7f82541439b0 2024-03-29T11:15:31.243 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.252+0000 7f823bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:31.243 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.253+0000 7f823bfff640 1 -- 172.21.15.170:0/658446904 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466496474 0 0) 0x7f82541274d0 con 0x7f82541439b0 2024-03-29T11:15:31.243 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.253+0000 7f823bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:31.243 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.253+0000 7f823bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:31.243 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.253+0000 7f823bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:31.243 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.253+0000 7f823bfff640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:15:31.243 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.253+0000 7f823bfff640 1 -- 172.21.15.170:0/658446904 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8240004eb0 con 0x7f82541439b0 2024-03-29T11:15:31.244 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.253+0000 7f823bfff640 10 client.?.objecter ms_handle_connect 0x7f82541470b0 2024-03-29T11:15:31.244 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.253+0000 7f823bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:31.244 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.253+0000 7f823bfff640 10 client.?.objecter ms_handle_connect 0x7f82540b4330 2024-03-29T11:15:31.244 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.253+0000 7f823bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:31.244 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.253+0000 7f823bfff640 1 -- 172.21.15.170:0/658446904 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958093251 0 0) 0x7f8254122900 con 0x7f82541470b0 2024-03-29T11:15:31.244 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.253+0000 7f823bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:31.244 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.253+0000 7f823bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:31.244 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.253+0000 7f823bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:31.244 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.253+0000 7f823bfff640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:15:31.244 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.253+0000 7f823bfff640 1 -- 172.21.15.170:0/658446904 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82541274d0 con 0x7f82541470b0 2024-03-29T11:15:31.244 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.253+0000 7f823bfff640 1 -- 172.21.15.170:0/658446904 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4282405612 0 0) 0x7f8240004eb0 con 0x7f82541439b0 2024-03-29T11:15:31.245 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f823bfff640 1 -- 172.21.15.170:0/658446904 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8254122900 con 0x7f82541439b0 2024-03-29T11:15:31.245 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f823bfff640 1 -- 172.21.15.170:0/658446904 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f8244001970 con 0x7f82541439b0 2024-03-29T11:15:31.245 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f823bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:31.245 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f823bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:31.245 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f823bfff640 10 monclient(hunting): dump: 2024-03-29T11:15:31.245 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:31.245 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:31.245 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:31.245 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:31.245 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:31.245 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:31.245 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:31.245 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:31.245 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:31.245 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:31.246 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f823bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:31.246 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f823bfff640 1 -- 172.21.15.170:0/658446904 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4040136906 0 0) 0x7f8254122900 con 0x7f82541439b0 2024-03-29T11:15:31.246 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f823bfff640 1 -- 172.21.15.170:0/658446904 >> v1:172.21.15.170:6790/0 conn(0x7f82540b4330 legacy=0x7f8254141090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:31.246 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f823bfff640 1 -- 172.21.15.170:0/658446904 >> v1:172.21.15.170:6789/0 conn(0x7f82541470b0 legacy=0x7f82541e6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:31.246 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f823bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:31.246 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f823bfff640 1 monclient: found mon.b 2024-03-29T11:15:31.246 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f823bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:31.246 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f823bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:15:31.246 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f823bfff640 1 -- 172.21.15.170:0/658446904 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82541266d0 con 0x7f82541439b0 2024-03-29T11:15:31.246 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f823bfff640 10 monclient: _finish_auth 0 2024-03-29T11:15:31.246 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f823bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:31.246 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f823bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:31.246 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f8259a63640 5 monclient: authenticate success, global_id 4357 2024-03-29T11:15:31.246 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f8259a63640 10 monclient: _renew_subs 2024-03-29T11:15:31.247 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f8259a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:15:31.247 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.254+0000 7f8259a63640 1 -- 172.21.15.170:0/658446904 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82541250e0 con 0x7f82541439b0 2024-03-29T11:15:31.247 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.255+0000 7f8259a63640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:31.247 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.255+0000 7f8259a63640 10 monclient: _renew_subs 2024-03-29T11:15:31.247 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.255+0000 7f8259a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:15:31.247 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.255+0000 7f8259a63640 1 -- 172.21.15.170:0/658446904 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8254125c10 con 0x7f82541439b0 2024-03-29T11:15:31.247 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.255+0000 7f8259a63640 1 librados: init done 2024-03-29T11:15:31.247 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.255+0000 7f823bfff640 1 -- 172.21.15.170:0/658446904 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82440047e0 con 0x7f82541439b0 2024-03-29T11:15:31.247 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.255+0000 7f823bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:31.247 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.255+0000 7f823bfff640 1 -- 172.21.15.170:0/658446904 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f8244004af0 con 0x7f82541439b0 2024-03-29T11:15:31.247 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.255+0000 7f823bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:31.247 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.255+0000 7f823bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:31.247 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.255+0000 7f823bfff640 10 monclient: dump: 2024-03-29T11:15:31.247 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:31.247 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:31.248 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:31.248 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:31.248 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:31.248 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:31.248 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:31.248 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:31.248 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:31.248 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:31.248 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.255+0000 7f82517fa640 4 set_mon_vals no callback set 2024-03-29T11:15:31.248 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.256+0000 7f8259a63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:31.248 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.256+0000 7f8259a63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:31.248 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.256+0000 7f8259a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:15:31.248 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.256+0000 7f8259a63640 1 -- 172.21.15.170:0/658446904 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8234005080 con 0x7f82541439b0 2024-03-29T11:15:31.248 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.257+0000 7f823bfff640 1 -- 172.21.15.170:0/658446904 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f8244004c50 con 0x7f82541439b0 2024-03-29T11:15:31.249 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.258+0000 7f823bfff640 1 -- 172.21.15.170:0/658446904 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f8244084a90 con 0x7f82541439b0 2024-03-29T11:15:31.249 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.258+0000 7f823bfff640 10 client.4357.objecter ms_dispatch 0x7f8254000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:31.249 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.258+0000 7f823bfff640 3 client.4357.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:31.249 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.258+0000 7f823bfff640 3 client.4357.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:31.249 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.258+0000 7f823bfff640 20 client.4357.objecter dump_active .. 0 homeless 2024-03-29T11:15:31.249 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.258+0000 7f82537fe640 10 monclient: get_auth_request con 0x7f824006db00 auth_method 0 2024-03-29T11:15:31.249 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.259+0000 7f823bfff640 10 client.4357.objecter ms_handle_connect 0x7f824006db00 2024-03-29T11:15:31.250 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.266+0000 7f823bfff640 1 -- 172.21.15.170:0/658446904 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8244050370 con 0x7f82541439b0 2024-03-29T11:15:31.250 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.266+0000 7f823bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:31.250 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.266+0000 7f823bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:31.613 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.636+0000 7f8259a63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:31.614 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.636+0000 7f8259a63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:31.614 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.636+0000 7f8259a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:15:31.614 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.636+0000 7f8259a63640 1 -- 172.21.15.170:0/658446904 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8234005330 con 0x7f82541439b0 2024-03-29T11:15:31.615 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.637+0000 7f823bfff640 1 -- 172.21.15.170:0/658446904 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f82440522a0 con 0x7f82541439b0 2024-03-29T11:15:31.615 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.637+0000 7f823bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:31.615 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.637+0000 7f823bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:31.616 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:15:31.616 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:31.619 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.641+0000 7f8259a63640 10 librados: watch_flush enter 2024-03-29T11:15:31.619 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.641+0000 7f8259a63640 10 librados: watch_flush exit 2024-03-29T11:15:31.619 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.641+0000 7f8259a63640 20 client.4357.objecter shutdown clearing up homeless session... 2024-03-29T11:15:31.619 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.641+0000 7f8259a63640 10 client.4357.objecter successfully canceled tick 2024-03-29T11:15:31.619 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.641+0000 7f8259a63640 1 -- 172.21.15.170:0/658446904 >> v1:172.21.15.170:6816/3298994444 conn(0x7f824006db00 legacy=0x7f824006ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:31.619 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.641+0000 7f8259a63640 10 monclient: shutdown 2024-03-29T11:15:31.619 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.641+0000 7f8259a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:31.619 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.641+0000 7f8259a63640 1 -- 172.21.15.170:0/658446904 >> v1:172.21.15.184:6789/0 conn(0x7f82541439b0 legacy=0x7f82541db9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:31.620 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.641+0000 7f8259a63640 1 -- 172.21.15.170:0/658446904 shutdown_connections 2024-03-29T11:15:31.620 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.641+0000 7f8259a63640 1 -- 172.21.15.170:0/658446904 >> 172.21.15.170:0/658446904 conn(0x7f8254066a00 msgr2=0x7f82540b4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:31.620 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.641+0000 7f8259a63640 1 -- 172.21.15.170:0/658446904 shutdown_connections 2024-03-29T11:15:31.620 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.641+0000 7f8259a63640 1 -- 172.21.15.170:0/658446904 wait complete. 2024-03-29T11:15:31.620 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:31.642+0000 7f8259a63640 1 librados: shutdown 2024-03-29T11:15:31.635 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:38.443 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:38.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.615+0000 7fe6f2e84640 10 monclient: get_monmap_and_config 2024-03-29T11:15:38.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.615+0000 7fe6f2e84640 10 monclient: build_initial_monmap 2024-03-29T11:15:38.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.615+0000 7fe6f2e84640 10 monclient: monmap: 2024-03-29T11:15:38.593 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:15:38.593 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:38.593 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:15:38.616388+0000 2024-03-29T11:15:38.593 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:15:38.616388+0000 2024-03-29T11:15:38.593 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:38.593 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:38.593 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:15:38.593 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:15:38.594 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:15:38.594 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:38.595 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.618+0000 7fe6f2e84640 1 Processor -- start 2024-03-29T11:15:38.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.619+0000 7fe6f2e84640 1 -- start start 2024-03-29T11:15:38.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.619+0000 7fe6f2e84640 10 monclient: init 2024-03-29T11:15:38.597 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.620+0000 7fe6f2e84640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:38.597 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.620+0000 7fe6f2e84640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:15:38.598 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.620+0000 7fe6f2e84640 10 monclient(hunting): picked mon.noname-c con 0x7fe6e40389e0 addr v1:172.21.15.184:6789/0 2024-03-29T11:15:38.598 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.620+0000 7fe6f2e84640 10 monclient(hunting): picked mon.noname-a con 0x7fe6e4038dc0 addr v1:172.21.15.170:6789/0 2024-03-29T11:15:38.598 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.620+0000 7fe6f2e84640 10 monclient(hunting): picked mon.noname-b con 0x7fe6e4030660 addr v1:172.21.15.170:6790/0 2024-03-29T11:15:38.598 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.620+0000 7fe6f2e84640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e40c54c0 con 0x7fe6e4038dc0 2024-03-29T11:15:38.598 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.620+0000 7fe6f2e84640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e4036430 con 0x7fe6e40389e0 2024-03-29T11:15:38.598 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.620+0000 7fe6f2e84640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e4039840 con 0x7fe6e4030660 2024-03-29T11:15:38.598 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.620+0000 7fe6f2e84640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:38.598 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.620+0000 7fe6f2e84640 10 monclient(hunting): authenticate will time out at 1119.931309s 2024-03-29T11:15:38.600 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.623+0000 7fe6f2683640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fe6e4030660 0x7fe6e40d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:39786/0 (socket says 172.21.15.170:39786) 2024-03-29T11:15:38.600 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.623+0000 7fe6f2683640 1 -- 172.21.15.170:0/869999193 learned_addr learned my addr 172.21.15.170:0/869999193 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:15:38.606 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.625+0000 7fe6f0e80640 1 -- 172.21.15.170:0/869999193 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485518490 0 0) 0x7fe6e4039840 con 0x7fe6e4030660 2024-03-29T11:15:38.606 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.626+0000 7fe6f0e80640 10 monclient(hunting): _negotiate 2024-03-29T11:15:38.606 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.626+0000 7fe6f0e80640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:38.606 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.626+0000 7fe6f0e80640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:38.606 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.626+0000 7fe6f0e80640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:15:38.606 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.626+0000 7fe6f0e80640 1 -- 172.21.15.170:0/869999193 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6d8003520 con 0x7fe6e4030660 2024-03-29T11:15:38.606 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.626+0000 7fe6f0e80640 1 -- 172.21.15.170:0/869999193 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941764194 0 0) 0x7fe6e4036430 con 0x7fe6e40389e0 2024-03-29T11:15:38.606 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.626+0000 7fe6f0e80640 10 monclient(hunting): _negotiate 2024-03-29T11:15:38.606 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.626+0000 7fe6f0e80640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:38.606 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.626+0000 7fe6f0e80640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:38.606 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.626+0000 7fe6f0e80640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:15:38.606 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.626+0000 7fe6f0e80640 1 -- 172.21.15.170:0/869999193 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6e4039840 con 0x7fe6e40389e0 2024-03-29T11:15:38.606 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.626+0000 7fe6f0e80640 1 -- 172.21.15.170:0/869999193 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238201758 0 0) 0x7fe6e40c54c0 con 0x7fe6e4038dc0 2024-03-29T11:15:38.607 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.626+0000 7fe6f0e80640 10 monclient(hunting): _negotiate 2024-03-29T11:15:38.607 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.626+0000 7fe6f0e80640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:38.607 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.626+0000 7fe6f0e80640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:38.607 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.626+0000 7fe6f0e80640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:15:38.607 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.627+0000 7fe6f0e80640 1 -- 172.21.15.170:0/869999193 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6e4036430 con 0x7fe6e4038dc0 2024-03-29T11:15:38.607 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.627+0000 7fe6f0e80640 1 -- 172.21.15.170:0/869999193 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3509717711 0 0) 0x7fe6d8003520 con 0x7fe6e4030660 2024-03-29T11:15:38.607 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.627+0000 7fe6f0e80640 1 -- 172.21.15.170:0/869999193 >> v1:172.21.15.184:6789/0 conn(0x7fe6e40389e0 legacy=0x7fe6e4033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:38.607 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.627+0000 7fe6f0e80640 1 -- 172.21.15.170:0/869999193 >> v1:172.21.15.170:6789/0 conn(0x7fe6e4038dc0 legacy=0x7fe6e40cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:38.607 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.627+0000 7fe6f0e80640 10 monclient: _finish_hunting 0 2024-03-29T11:15:38.607 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.627+0000 7fe6f0e80640 1 monclient: found mon.noname-b 2024-03-29T11:15:38.607 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.627+0000 7fe6f0e80640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:38.607 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.627+0000 7fe6f0e80640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:15:38.607 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.628+0000 7fe6f0e80640 1 -- 172.21.15.170:0/869999193 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e4035690 con 0x7fe6e4030660 2024-03-29T11:15:38.607 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.628+0000 7fe6f0e80640 10 monclient: _finish_auth 0 2024-03-29T11:15:38.608 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.628+0000 7fe6f0e80640 10 monclient: _check_auth_tickets 2024-03-29T11:15:38.608 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.628+0000 7fe6f0e80640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:38.608 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.628+0000 7fe6f2e84640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:15:38.608 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.628+0000 7fe6f2e84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:38.608 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.628+0000 7fe6f0e80640 1 -- 172.21.15.170:0/869999193 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe6ec04f730 con 0x7fe6e4030660 2024-03-29T11:15:38.608 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.628+0000 7fe6f0e80640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:38.608 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.628+0000 7fe6f0e80640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:38.608 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.628+0000 7fe6f0e80640 10 monclient: dump: 2024-03-29T11:15:38.608 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:38.608 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:38.608 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:38.608 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:38.608 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:38.608 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:38.608 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:38.609 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:38.609 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:38.609 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:38.609 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.628+0000 7fe6f2e84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:38.610 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.630+0000 7fe6f0e80640 1 -- 172.21.15.170:0/869999193 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6ec0008d0 con 0x7fe6e4030660 2024-03-29T11:15:38.610 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.630+0000 7fe6f0e80640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:38.610 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.630+0000 7fe6f0e80640 1 -- 172.21.15.170:0/869999193 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe6ec0758b0 con 0x7fe6e4030660 2024-03-29T11:15:38.610 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.630+0000 7fe6f0e80640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:38.610 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.630+0000 7fe6f0e80640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:38.610 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.630+0000 7fe6f0e80640 10 monclient: dump: 2024-03-29T11:15:38.610 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:38.610 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:38.610 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:38.610 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:38.611 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:38.611 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:38.611 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:38.611 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:38.611 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:38.611 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:38.611 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.632+0000 7fe6f2e84640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:38.611 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.632+0000 7fe6f2e84640 4 set_mon_vals no callback set 2024-03-29T11:15:38.611 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.633+0000 7fe6f2e84640 10 monclient: shutdown 2024-03-29T11:15:38.611 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.633+0000 7fe6f2e84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:38.611 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.633+0000 7fe6f2e84640 1 -- 172.21.15.170:0/869999193 >> v1:172.21.15.170:6790/0 conn(0x7fe6e4030660 legacy=0x7fe6e40d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:38.611 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.633+0000 7fe6f2e84640 1 -- 172.21.15.170:0/869999193 shutdown_connections 2024-03-29T11:15:38.611 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.633+0000 7fe6f2e84640 1 -- 172.21.15.170:0/869999193 >> 172.21.15.170:0/869999193 conn(0x7fe6e4005c00 msgr2=0x7fe6e4006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:38.612 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.634+0000 7fe6f2e84640 1 -- 172.21.15.170:0/869999193 shutdown_connections 2024-03-29T11:15:38.612 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.634+0000 7fe6f2e84640 1 -- 172.21.15.170:0/869999193 wait complete. 2024-03-29T11:15:38.612 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.635+0000 7fe6f2e84640 10 monclient: build_initial_monmap 2024-03-29T11:15:38.612 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.635+0000 7fe6f2e84640 10 monclient: monmap: 2024-03-29T11:15:38.612 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:15:38.613 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:38.613 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:15:38.613 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:15:38.613 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:38.613 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:38.613 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:15:38.613 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:15:38.613 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:15:38.613 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:38.613 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.635+0000 7fe6f2e84640 1 librados: starting msgr at 2024-03-29T11:15:38.613 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.635+0000 7fe6f2e84640 1 librados: starting objecter 2024-03-29T11:15:38.613 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.635+0000 7fe6f2e84640 1 Processor -- start 2024-03-29T11:15:38.614 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.636+0000 7fe6f2e84640 1 -- start start 2024-03-29T11:15:38.614 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.636+0000 7fe6f2e84640 1 librados: setting wanted keys 2024-03-29T11:15:38.614 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.636+0000 7fe6f2e84640 1 librados: calling monclient init 2024-03-29T11:15:38.614 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.636+0000 7fe6f2e84640 10 monclient: init 2024-03-29T11:15:38.614 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.636+0000 7fe6f2e84640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:38.615 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.636+0000 7fe6f2e84640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:15:38.615 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.636+0000 7fe6f2e84640 10 monclient(hunting): picked mon.noname-a con 0x7fe6e4030660 addr v1:172.21.15.170:6789/0 2024-03-29T11:15:38.615 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.636+0000 7fe6f2e84640 10 monclient(hunting): picked mon.noname-c con 0x7fe6e40389e0 addr v1:172.21.15.170:6790/0 2024-03-29T11:15:38.615 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.636+0000 7fe6f2e84640 10 monclient(hunting): picked mon.noname-b con 0x7fe6e4161940 addr v1:172.21.15.184:6789/0 2024-03-29T11:15:38.615 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.636+0000 7fe6f2e84640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e4160da0 con 0x7fe6e4030660 2024-03-29T11:15:38.615 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.636+0000 7fe6f2e84640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e402e3b0 con 0x7fe6e4161940 2024-03-29T11:15:38.615 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.636+0000 7fe6f2e84640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e402e620 con 0x7fe6e40389e0 2024-03-29T11:15:38.615 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.636+0000 7fe6f2e84640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:38.615 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.636+0000 7fe6f2e84640 10 monclient(hunting): authenticate will time out at 1119.947937s 2024-03-29T11:15:38.615 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.637+0000 7fe6f1681640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fe6e40389e0 0x7fe6e4161230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:40344/0 (socket says 172.21.15.170:40344) 2024-03-29T11:15:38.615 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.637+0000 7fe6f1681640 1 -- 172.21.15.170:0/3727464683 learned_addr learned my addr 172.21.15.170:0/3727464683 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:15:38.615 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.637+0000 7fe6e27fc640 10 client.?.objecter ms_handle_connect 0x7fe6e40389e0 2024-03-29T11:15:38.615 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.637+0000 7fe6e27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:38.616 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.638+0000 7fe6e27fc640 1 -- 172.21.15.170:0/3727464683 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986738235 0 0) 0x7fe6e402e620 con 0x7fe6e40389e0 2024-03-29T11:15:38.616 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.638+0000 7fe6e27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:38.616 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.638+0000 7fe6e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:38.616 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.638+0000 7fe6e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:38.616 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.638+0000 7fe6e27fc640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:15:38.616 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.638+0000 7fe6e27fc640 1 -- 172.21.15.170:0/3727464683 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6d0004eb0 con 0x7fe6e40389e0 2024-03-29T11:15:38.616 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.639+0000 7fe6e27fc640 1 -- 172.21.15.170:0/3727464683 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2526723516 0 0) 0x7fe6d0004eb0 con 0x7fe6e40389e0 2024-03-29T11:15:38.616 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.639+0000 7fe6e27fc640 1 -- 172.21.15.170:0/3727464683 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6e402e620 con 0x7fe6e40389e0 2024-03-29T11:15:38.616 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.639+0000 7fe6e27fc640 1 -- 172.21.15.170:0/3727464683 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe6dc001970 con 0x7fe6e40389e0 2024-03-29T11:15:38.617 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.639+0000 7fe6e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:38.617 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.639+0000 7fe6e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:38.617 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.639+0000 7fe6e27fc640 10 monclient(hunting): dump: 2024-03-29T11:15:38.617 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:38.617 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:38.617 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:38.617 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:38.617 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:38.617 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:38.617 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:38.617 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:38.617 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:38.617 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:38.617 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.639+0000 7fe6e27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:38.618 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.640+0000 7fe6e27fc640 1 -- 172.21.15.170:0/3727464683 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3333824527 0 0) 0x7fe6e402e620 con 0x7fe6e40389e0 2024-03-29T11:15:38.618 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.640+0000 7fe6e27fc640 1 -- 172.21.15.170:0/3727464683 >> v1:172.21.15.184:6789/0 conn(0x7fe6e4161940 legacy=0x7fe6e416b7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:15:38.620 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6e27fc640 1 -- 172.21.15.170:0/3727464683 >> v1:172.21.15.170:6789/0 conn(0x7fe6e4030660 legacy=0x7fe6e402d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:38.620 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6e27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:15:38.620 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6e27fc640 1 monclient: found mon.c 2024-03-29T11:15:38.620 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:38.620 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6e27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:15:38.620 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6e27fc640 1 -- 172.21.15.170:0/3727464683 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e402e900 con 0x7fe6e40389e0 2024-03-29T11:15:38.620 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6e27fc640 10 monclient: _finish_auth 0 2024-03-29T11:15:38.620 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6e27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:15:38.620 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:38.620 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6e27fc640 10 client.?.objecter ms_handle_connect 0x7fe6e4030660 2024-03-29T11:15:38.620 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6e27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:38.621 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6f2e84640 5 monclient: authenticate success, global_id 4388 2024-03-29T11:15:38.621 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6f2e84640 10 monclient: _renew_subs 2024-03-29T11:15:38.621 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6f2e84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:15:38.621 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6f2e84640 1 -- 172.21.15.170:0/3727464683 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e40c0c20 con 0x7fe6e40389e0 2024-03-29T11:15:38.621 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6f2e84640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:38.621 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6f2e84640 10 monclient: _renew_subs 2024-03-29T11:15:38.621 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6f2e84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:15:38.621 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6f2e84640 1 -- 172.21.15.170:0/3727464683 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e416cf00 con 0x7fe6e40389e0 2024-03-29T11:15:38.621 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.643+0000 7fe6f2e84640 1 librados: init done 2024-03-29T11:15:38.622 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.645+0000 7fe6e27fc640 1 -- 172.21.15.170:0/3727464683 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6dc001bd0 con 0x7fe6e40389e0 2024-03-29T11:15:38.622 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.645+0000 7fe6e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:38.622 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.645+0000 7fe6e27fc640 1 -- 172.21.15.170:0/3727464683 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe6dc003a60 con 0x7fe6e40389e0 2024-03-29T11:15:38.622 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.645+0000 7fe6e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:38.622 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.645+0000 7fe6e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:38.622 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.645+0000 7fe6e27fc640 10 monclient: dump: 2024-03-29T11:15:38.622 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:38.623 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:38.623 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:38.623 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:38.623 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:38.623 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:38.623 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:38.623 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:38.623 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:38.623 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:38.623 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.645+0000 7fe6f2e84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:38.623 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.645+0000 7fe6f2e84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:38.623 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.645+0000 7fe6f2e84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:15:38.623 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.645+0000 7fe6f2e84640 1 -- 172.21.15.170:0/3727464683 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e4007b90 con 0x7fe6e40389e0 2024-03-29T11:15:38.623 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.645+0000 7fe6e3fff640 4 set_mon_vals no callback set 2024-03-29T11:15:38.625 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.647+0000 7fe6e27fc640 1 -- 172.21.15.170:0/3727464683 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fe6dc003bc0 con 0x7fe6e40389e0 2024-03-29T11:15:38.625 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.648+0000 7fe6e27fc640 1 -- 172.21.15.170:0/3727464683 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fe6dc084b60 con 0x7fe6e40389e0 2024-03-29T11:15:38.625 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.648+0000 7fe6e27fc640 10 client.4388.objecter ms_dispatch 0x7fe6ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:38.626 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.648+0000 7fe6e27fc640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:38.626 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.648+0000 7fe6e27fc640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:38.626 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.648+0000 7fe6e27fc640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-29T11:15:38.626 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.648+0000 7fe6f1e82640 10 monclient: get_auth_request con 0x7fe6d006dae0 auth_method 0 2024-03-29T11:15:38.626 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.649+0000 7fe6e27fc640 10 client.4388.objecter ms_handle_connect 0x7fe6d006dae0 2024-03-29T11:15:38.631 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.654+0000 7fe6e27fc640 1 -- 172.21.15.170:0/3727464683 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe6dc050440 con 0x7fe6e40389e0 2024-03-29T11:15:38.631 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.654+0000 7fe6e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:38.631 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.654+0000 7fe6e27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:38.963 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.986+0000 7fe6f2e84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:38.963 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.986+0000 7fe6f2e84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:38.963 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.986+0000 7fe6f2e84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:15:38.964 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.986+0000 7fe6f2e84640 1 -- 172.21.15.170:0/3727464683 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6e4005120 con 0x7fe6e40389e0 2024-03-29T11:15:38.964 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.987+0000 7fe6e27fc640 1 -- 172.21.15.170:0/3727464683 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fe6dc052370 con 0x7fe6e40389e0 2024-03-29T11:15:38.964 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.987+0000 7fe6e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:38.964 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.987+0000 7fe6e27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:38.964 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:15:38.965 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:38.967 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.990+0000 7fe6f2e84640 10 librados: watch_flush enter 2024-03-29T11:15:38.967 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.990+0000 7fe6f2e84640 10 librados: watch_flush exit 2024-03-29T11:15:38.967 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.990+0000 7fe6f2e84640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-29T11:15:38.967 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.990+0000 7fe6f2e84640 10 client.4388.objecter successfully canceled tick 2024-03-29T11:15:38.967 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.990+0000 7fe6f2e84640 1 -- 172.21.15.170:0/3727464683 >> v1:172.21.15.170:6816/3298994444 conn(0x7fe6d006dae0 legacy=0x7fe6d006ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:38.967 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.990+0000 7fe6f2e84640 10 monclient: shutdown 2024-03-29T11:15:38.967 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.990+0000 7fe6f2e84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:38.967 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.990+0000 7fe6f2e84640 1 -- 172.21.15.170:0/3727464683 >> v1:172.21.15.170:6790/0 conn(0x7fe6e40389e0 legacy=0x7fe6e4161230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:38.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.990+0000 7fe6f2e84640 1 -- 172.21.15.170:0/3727464683 shutdown_connections 2024-03-29T11:15:38.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.990+0000 7fe6f2e84640 1 -- 172.21.15.170:0/3727464683 >> 172.21.15.170:0/3727464683 conn(0x7fe6e4005c00 msgr2=0x7fe6e4034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:38.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.991+0000 7fe6f2e84640 1 -- 172.21.15.170:0/3727464683 shutdown_connections 2024-03-29T11:15:38.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.991+0000 7fe6f2e84640 1 -- 172.21.15.170:0/3727464683 wait complete. 2024-03-29T11:15:38.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:38.991+0000 7fe6f2e84640 1 librados: shutdown 2024-03-29T11:15:38.981 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:45.789 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:45.942 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.962+0000 7f8700fb5640 10 monclient: get_monmap_and_config 2024-03-29T11:15:45.942 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.962+0000 7f8700fb5640 10 monclient: build_initial_monmap 2024-03-29T11:15:45.942 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.962+0000 7f8700fb5640 10 monclient: monmap: 2024-03-29T11:15:45.942 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:15:45.942 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:45.942 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:15:45.963621+0000 2024-03-29T11:15:45.942 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:15:45.963621+0000 2024-03-29T11:15:45.942 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:45.942 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:45.942 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:15:45.943 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:15:45.943 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:15:45.943 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:45.943 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.963+0000 7f8700fb5640 1 Processor -- start 2024-03-29T11:15:45.943 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.963+0000 7f8700fb5640 1 -- start start 2024-03-29T11:15:45.943 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.963+0000 7f8700fb5640 10 monclient: init 2024-03-29T11:15:45.943 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.964+0000 7f8700fb5640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:45.943 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.964+0000 7f8700fb5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:15:45.943 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.964+0000 7f8700fb5640 10 monclient(hunting): picked mon.noname-a con 0x7f86fc0af6c0 addr v1:172.21.15.170:6789/0 2024-03-29T11:15:45.943 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.964+0000 7f8700fb5640 10 monclient(hunting): picked mon.noname-b con 0x7f86fc14fdc0 addr v1:172.21.15.170:6790/0 2024-03-29T11:15:45.943 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.964+0000 7f8700fb5640 10 monclient(hunting): picked mon.noname-c con 0x7f86fc153090 addr v1:172.21.15.184:6789/0 2024-03-29T11:15:45.943 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.964+0000 7f8700fb5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86fc0a94e0 con 0x7f86fc0af6c0 2024-03-29T11:15:45.943 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.964+0000 7f8700fb5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86fc0af3a0 con 0x7f86fc153090 2024-03-29T11:15:45.943 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.964+0000 7f8700fb5640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86fc0afb60 con 0x7f86fc14fdc0 2024-03-29T11:15:45.943 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.964+0000 7f8700fb5640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:45.944 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.964+0000 7f8700fb5640 10 monclient(hunting): authenticate will time out at 1127.276323s 2024-03-29T11:15:45.945 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.967+0000 7f86fad9d640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f86fc153090 0x7f86fc1538a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:49753/0 (socket says 172.21.15.170:49753) 2024-03-29T11:15:45.945 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.967+0000 7f86fad9d640 1 -- 172.21.15.170:0/2387380318 learned_addr learned my addr 172.21.15.170:0/2387380318 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:15:45.946 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.968+0000 7f86f959a640 1 -- 172.21.15.170:0/2387380318 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190458825 0 0) 0x7f86fc0af3a0 con 0x7f86fc153090 2024-03-29T11:15:45.946 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.968+0000 7f86f959a640 10 monclient(hunting): _negotiate 2024-03-29T11:15:45.946 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.968+0000 7f86f959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:45.946 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.968+0000 7f86f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:45.946 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.968+0000 7f86f959a640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:15:45.946 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.968+0000 7f86f959a640 1 -- 172.21.15.170:0/2387380318 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e4003520 con 0x7f86fc153090 2024-03-29T11:15:45.947 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.969+0000 7f86f959a640 1 -- 172.21.15.170:0/2387380318 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3724483009 0 0) 0x7f86e4003520 con 0x7f86fc153090 2024-03-29T11:15:45.947 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.969+0000 7f86f959a640 1 -- 172.21.15.170:0/2387380318 >> v1:172.21.15.170:6790/0 conn(0x7f86fc14fdc0 legacy=0x7f86fc152980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:45.947 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.969+0000 7f86f959a640 1 -- 172.21.15.170:0/2387380318 >> v1:172.21.15.170:6789/0 conn(0x7f86fc0af6c0 legacy=0x7f86fc14f6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:45.948 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.970+0000 7f86f959a640 10 monclient: _finish_hunting 0 2024-03-29T11:15:45.948 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.970+0000 7f86f959a640 1 monclient: found mon.noname-c 2024-03-29T11:15:45.948 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.970+0000 7f86f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:45.948 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.970+0000 7f86f959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:15:45.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.970+0000 7f86f959a640 1 -- 172.21.15.170:0/2387380318 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86fc0aab80 con 0x7f86fc153090 2024-03-29T11:15:45.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.970+0000 7f86f959a640 10 monclient: _finish_auth 0 2024-03-29T11:15:45.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.970+0000 7f86f959a640 10 monclient: _check_auth_tickets 2024-03-29T11:15:45.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.970+0000 7f86f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:45.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.970+0000 7f86f959a640 1 -- 172.21.15.170:0/2387380318 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f86f4001a10 con 0x7f86fc153090 2024-03-29T11:15:45.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.970+0000 7f86f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:45.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.970+0000 7f86f959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:45.949 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.970+0000 7f86f959a640 10 monclient: dump: 2024-03-29T11:15:45.949 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:45.949 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:45.949 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:45.949 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:45.949 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:45.949 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:45.949 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:45.950 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:45.950 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:45.950 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:45.950 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.970+0000 7f8700fb5640 5 monclient: authenticate success, global_id 4363 2024-03-29T11:15:45.950 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.970+0000 7f8700fb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:45.950 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.970+0000 7f86f959a640 1 -- 172.21.15.170:0/2387380318 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86f4001be0 con 0x7f86fc153090 2024-03-29T11:15:45.950 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.970+0000 7f86f959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:45.950 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.970+0000 7f8700fb5640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:45.950 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.970+0000 7f8700fb5640 4 set_mon_vals no callback set 2024-03-29T11:15:45.951 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.971+0000 7f8700fb5640 10 monclient: shutdown 2024-03-29T11:15:45.951 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.971+0000 7f8700fb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:45.951 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.971+0000 7f8700fb5640 1 -- 172.21.15.170:0/2387380318 >> v1:172.21.15.184:6789/0 conn(0x7f86fc153090 legacy=0x7f86fc1538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:45.951 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.971+0000 7f8700fb5640 1 -- 172.21.15.170:0/2387380318 shutdown_connections 2024-03-29T11:15:45.951 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.971+0000 7f8700fb5640 1 -- 172.21.15.170:0/2387380318 >> 172.21.15.170:0/2387380318 conn(0x7f86fc0b35e0 msgr2=0x7f86fc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:45.951 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.973+0000 7f8700fb5640 1 -- 172.21.15.170:0/2387380318 shutdown_connections 2024-03-29T11:15:45.952 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.974+0000 7f8700fb5640 1 -- 172.21.15.170:0/2387380318 wait complete. 2024-03-29T11:15:45.955 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.975+0000 7f8700fb5640 10 monclient: build_initial_monmap 2024-03-29T11:15:45.955 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.975+0000 7f8700fb5640 10 monclient: monmap: 2024-03-29T11:15:45.955 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:15:45.955 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:45.955 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:15:45.955 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:15:45.956 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:45.956 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:45.956 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:15:45.956 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:15:45.956 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:15:45.956 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:45.956 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.975+0000 7f8700fb5640 1 librados: starting msgr at 2024-03-29T11:15:45.956 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.975+0000 7f8700fb5640 1 librados: starting objecter 2024-03-29T11:15:45.956 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.975+0000 7f8700fb5640 1 Processor -- start 2024-03-29T11:15:45.956 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.975+0000 7f8700fb5640 1 -- start start 2024-03-29T11:15:45.956 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.975+0000 7f8700fb5640 1 librados: setting wanted keys 2024-03-29T11:15:45.956 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.975+0000 7f8700fb5640 1 librados: calling monclient init 2024-03-29T11:15:45.956 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.975+0000 7f8700fb5640 10 monclient: init 2024-03-29T11:15:45.956 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.976+0000 7f8700fb5640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:45.956 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.976+0000 7f8700fb5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:45.957 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.976+0000 7f8700fb5640 10 monclient(hunting): picked mon.noname-a con 0x7f86fc0af6c0 addr v1:172.21.15.170:6789/0 2024-03-29T11:15:45.957 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.976+0000 7f8700fb5640 10 monclient(hunting): picked mon.noname-b con 0x7f86fc14fdc0 addr v1:172.21.15.184:6789/0 2024-03-29T11:15:45.957 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.976+0000 7f8700fb5640 10 monclient(hunting): picked mon.noname-c con 0x7f86fc153090 addr v1:172.21.15.170:6790/0 2024-03-29T11:15:45.957 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.976+0000 7f8700fb5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86fc164370 con 0x7f86fc0af6c0 2024-03-29T11:15:45.957 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.976+0000 7f8700fb5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86fc1645c0 con 0x7f86fc14fdc0 2024-03-29T11:15:45.957 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.976+0000 7f8700fb5640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86fc162f60 con 0x7f86fc153090 2024-03-29T11:15:45.957 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.976+0000 7f8700fb5640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:45.957 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.976+0000 7f8700fb5640 10 monclient(hunting): authenticate will time out at 1127.288547s 2024-03-29T11:15:45.965 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.982+0000 7f86fad9d640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f86fc153090 0x7f86fc1f8820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:55401/0 (socket says 172.21.15.170:55401) 2024-03-29T11:15:45.965 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.982+0000 7f86fad9d640 1 -- 172.21.15.170:0/156363318 learned_addr learned my addr 172.21.15.170:0/156363318 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:15:45.965 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.982+0000 7f86eaffd640 10 client.?.objecter ms_handle_connect 0x7f86fc153090 2024-03-29T11:15:45.965 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.982+0000 7f86eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:45.965 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.982+0000 7f86eaffd640 10 client.?.objecter ms_handle_connect 0x7f86fc0af6c0 2024-03-29T11:15:45.965 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.982+0000 7f86eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:45.965 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.982+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2374739775 0 0) 0x7f86fc164370 con 0x7f86fc0af6c0 2024-03-29T11:15:45.965 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.982+0000 7f86eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:15:45.965 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.982+0000 7f86eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:45.965 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.982+0000 7f86eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:45.965 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.982+0000 7f86eaffd640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:15:45.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.982+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86dc004eb0 con 0x7f86fc0af6c0 2024-03-29T11:15:45.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.982+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263191504 0 0) 0x7f86fc162f60 con 0x7f86fc153090 2024-03-29T11:15:45.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.982+0000 7f86eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:15:45.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.982+0000 7f86eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:45.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.982+0000 7f86eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:45.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.982+0000 7f86eaffd640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:15:45.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.982+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86dc0051d0 con 0x7f86fc153090 2024-03-29T11:15:45.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1807762932 0 0) 0x7f86dc004eb0 con 0x7f86fc0af6c0 2024-03-29T11:15:45.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86dc005640 con 0x7f86fc0af6c0 2024-03-29T11:15:45.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4201144718 0 0) 0x7f86dc0051d0 con 0x7f86fc153090 2024-03-29T11:15:45.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86dc004eb0 con 0x7f86fc153090 2024-03-29T11:15:45.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f86ec001830 con 0x7f86fc0af6c0 2024-03-29T11:15:45.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:45.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:45.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 10 monclient(hunting): dump: 2024-03-29T11:15:45.967 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:45.967 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:45.967 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:45.967 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:45.967 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:45.967 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:45.967 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:45.967 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:45.967 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:45.967 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:45.967 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:45.967 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f86f4004d00 con 0x7f86fc153090 2024-03-29T11:15:45.967 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:45.967 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:45.967 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 10 monclient(hunting): dump: 2024-03-29T11:15:45.968 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:45.968 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:45.968 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:45.968 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:45.968 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:45.968 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:45.968 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:45.968 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:45.968 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:45.968 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:45.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 729603240 0 0) 0x7f86dc005640 con 0x7f86fc0af6c0 2024-03-29T11:15:45.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 >> v1:172.21.15.170:6790/0 conn(0x7f86fc153090 legacy=0x7f86fc1f8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:45.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 >> v1:172.21.15.184:6789/0 conn(0x7f86fc14fdc0 legacy=0x7f86fc1f8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:45.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:15:45.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 1 monclient: found mon.a 2024-03-29T11:15:45.969 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:45.969 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:15:45.969 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86fc161f60 con 0x7f86fc0af6c0 2024-03-29T11:15:45.969 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 10 monclient: _finish_auth 0 2024-03-29T11:15:45.969 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:15:45.969 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:45.969 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f8700fb5640 5 monclient: authenticate success, global_id 4368 2024-03-29T11:15:45.969 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f8700fb5640 10 monclient: _renew_subs 2024-03-29T11:15:45.969 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f8700fb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:15:45.969 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f8700fb5640 1 -- 172.21.15.170:0/156363318 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86fc162150 con 0x7f86fc0af6c0 2024-03-29T11:15:45.969 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f8700fb5640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:45.969 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f8700fb5640 10 monclient: _renew_subs 2024-03-29T11:15:45.969 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f8700fb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:15:45.969 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f8700fb5640 1 -- 172.21.15.170:0/156363318 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86fc2007b0 con 0x7f86fc0af6c0 2024-03-29T11:15:45.970 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f8700fb5640 1 librados: init done 2024-03-29T11:15:45.970 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86ec0035f0 con 0x7f86fc0af6c0 2024-03-29T11:15:45.970 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:45.970 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f86ec003900 con 0x7f86fc0af6c0 2024-03-29T11:15:45.970 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:45.970 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:45.970 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86eaffd640 10 monclient: dump: 2024-03-29T11:15:45.970 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:45.970 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:45.970 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:45.970 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:45.970 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:45.970 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:45.970 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:45.971 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:45.971 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:45.971 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:45.971 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.983+0000 7f86ebfff640 4 set_mon_vals no callback set 2024-03-29T11:15:45.971 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.985+0000 7f8700fb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:45.971 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.985+0000 7f8700fb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:45.971 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.985+0000 7f8700fb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:15:45.971 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.985+0000 7f8700fb5640 1 -- 172.21.15.170:0/156363318 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e0005080 con 0x7f86fc0af6c0 2024-03-29T11:15:45.971 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.986+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f86ec0198d0 con 0x7f86fc0af6c0 2024-03-29T11:15:45.971 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.986+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f86ec083790 con 0x7f86fc0af6c0 2024-03-29T11:15:45.971 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.986+0000 7f86eaffd640 10 client.4368.objecter ms_dispatch 0x7f86fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:45.971 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.986+0000 7f86eaffd640 3 client.4368.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:45.971 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.986+0000 7f86eaffd640 3 client.4368.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:45.971 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.986+0000 7f86eaffd640 20 client.4368.objecter dump_active .. 0 homeless 2024-03-29T11:15:45.971 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.987+0000 7f86f9d9b640 10 monclient: get_auth_request con 0x7f86dc06e3a0 auth_method 0 2024-03-29T11:15:45.971 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.987+0000 7f86eaffd640 10 client.4368.objecter ms_handle_connect 0x7f86dc06e3a0 2024-03-29T11:15:45.972 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.994+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f86ec04eea0 con 0x7f86fc0af6c0 2024-03-29T11:15:45.973 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.994+0000 7f86eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:45.973 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:45.994+0000 7f86eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:46.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.322+0000 7f8700fb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:46.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.322+0000 7f8700fb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:46.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.322+0000 7f8700fb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:15:46.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.322+0000 7f8700fb5640 1 -- 172.21.15.170:0/156363318 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86e0005330 con 0x7f86fc0af6c0 2024-03-29T11:15:46.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.326+0000 7f86eaffd640 1 -- 172.21.15.170:0/156363318 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f86ec050fa0 con 0x7f86fc0af6c0 2024-03-29T11:15:46.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.326+0000 7f86eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:46.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.326+0000 7f86eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:46.304 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:15:46.304 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:46.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.329+0000 7f8700fb5640 10 librados: watch_flush enter 2024-03-29T11:15:46.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.329+0000 7f8700fb5640 10 librados: watch_flush exit 2024-03-29T11:15:46.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.329+0000 7f8700fb5640 20 client.4368.objecter shutdown clearing up homeless session... 2024-03-29T11:15:46.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.329+0000 7f8700fb5640 10 client.4368.objecter successfully canceled tick 2024-03-29T11:15:46.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.330+0000 7f8700fb5640 1 -- 172.21.15.170:0/156363318 >> v1:172.21.15.170:6816/3298994444 conn(0x7f86dc06e3a0 legacy=0x7f86dc070860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:46.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.330+0000 7f8700fb5640 10 monclient: shutdown 2024-03-29T11:15:46.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.330+0000 7f8700fb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:46.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.330+0000 7f8700fb5640 1 -- 172.21.15.170:0/156363318 >> v1:172.21.15.170:6789/0 conn(0x7f86fc0af6c0 legacy=0x7f86fc14e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:46.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.330+0000 7f8700fb5640 1 -- 172.21.15.170:0/156363318 shutdown_connections 2024-03-29T11:15:46.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.330+0000 7f8700fb5640 1 -- 172.21.15.170:0/156363318 >> 172.21.15.170:0/156363318 conn(0x7f86fc0b35e0 msgr2=0x7f86fc0a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:46.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.331+0000 7f8700fb5640 1 -- 172.21.15.170:0/156363318 shutdown_connections 2024-03-29T11:15:46.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.331+0000 7f8700fb5640 1 -- 172.21.15.170:0/156363318 wait complete. 2024-03-29T11:15:46.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:46.332+0000 7f8700fb5640 1 librados: shutdown 2024-03-29T11:15:46.328 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:53.137 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:53.282 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.304+0000 7f296d154640 10 monclient: get_monmap_and_config 2024-03-29T11:15:53.282 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.304+0000 7f296d154640 10 monclient: build_initial_monmap 2024-03-29T11:15:53.282 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.304+0000 7f296d154640 10 monclient: monmap: 2024-03-29T11:15:53.282 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:15:53.282 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:53.282 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:15:53.305492+0000 2024-03-29T11:15:53.282 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:15:53.305492+0000 2024-03-29T11:15:53.282 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:53.282 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:53.282 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:15:53.282 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:15:53.283 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:15:53.283 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:53.286 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.305+0000 7f296d154640 1 Processor -- start 2024-03-29T11:15:53.286 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.305+0000 7f296d154640 1 -- start start 2024-03-29T11:15:53.286 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.305+0000 7f296d154640 10 monclient: init 2024-03-29T11:15:53.287 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.306+0000 7f296d154640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:53.287 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.306+0000 7f296d154640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:15:53.287 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.306+0000 7f296d154640 10 monclient(hunting): picked mon.noname-c con 0x7f29680b1f60 addr v1:172.21.15.184:6789/0 2024-03-29T11:15:53.287 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.306+0000 7f296d154640 10 monclient(hunting): picked mon.noname-b con 0x7f2968071eb0 addr v1:172.21.15.170:6790/0 2024-03-29T11:15:53.287 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.306+0000 7f296d154640 10 monclient(hunting): picked mon.noname-a con 0x7f2968143d10 addr v1:172.21.15.170:6789/0 2024-03-29T11:15:53.287 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.306+0000 7f296d154640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296806ef80 con 0x7f2968143d10 2024-03-29T11:15:53.287 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.306+0000 7f296d154640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296806f260 con 0x7f29680b1f60 2024-03-29T11:15:53.287 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.306+0000 7f296d154640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29680b40c0 con 0x7f2968071eb0 2024-03-29T11:15:53.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.306+0000 7f296d154640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:53.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.306+0000 7f296d154640 10 monclient(hunting): authenticate will time out at 1134.618571s 2024-03-29T11:15:53.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.309+0000 7f296759e640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f2968143d10 0x7f2968146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:45377/0 (socket says 172.21.15.170:45377) 2024-03-29T11:15:53.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.309+0000 7f296759e640 1 -- 172.21.15.170:0/3976759396 learned_addr learned my addr 172.21.15.170:0/3976759396 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:15:53.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 1 -- 172.21.15.170:0/3976759396 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 238505861 0 0) 0x7f296806ef80 con 0x7f2968143d10 2024-03-29T11:15:53.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:15:53.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:53.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:53.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:15:53.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 1 -- 172.21.15.170:0/3976759396 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2950003520 con 0x7f2968143d10 2024-03-29T11:15:53.291 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 1 -- 172.21.15.170:0/3976759396 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148812834 0 0) 0x7f296806f260 con 0x7f29680b1f60 2024-03-29T11:15:53.291 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:15:53.291 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:53.291 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:53.292 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:15:53.292 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 1 -- 172.21.15.170:0/3976759396 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f296806ef80 con 0x7f29680b1f60 2024-03-29T11:15:53.292 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 1 -- 172.21.15.170:0/3976759396 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1637874224 0 0) 0x7f2950003520 con 0x7f2968143d10 2024-03-29T11:15:53.292 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 1 -- 172.21.15.170:0/3976759396 >> v1:172.21.15.170:6790/0 conn(0x7f2968071eb0 legacy=0x7f2968146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:53.292 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 1 -- 172.21.15.170:0/3976759396 >> v1:172.21.15.184:6789/0 conn(0x7f29680b1f60 legacy=0x7f2968143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:53.292 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:15:53.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 1 monclient: found mon.noname-a 2024-03-29T11:15:53.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:53.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:15:53.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 1 -- 172.21.15.170:0/3976759396 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2968155d70 con 0x7f2968143d10 2024-03-29T11:15:53.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 10 monclient: _finish_auth 0 2024-03-29T11:15:53.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:15:53.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:53.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 1 -- 172.21.15.170:0/3976759396 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f2960001990 con 0x7f2968143d10 2024-03-29T11:15:53.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.312+0000 7f2965d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:53.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.313+0000 7f2965d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:53.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.313+0000 7f2965d9b640 10 monclient: dump: 2024-03-29T11:15:53.294 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:53.294 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:53.294 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:53.295 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:53.295 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:53.295 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:53.295 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:53.295 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:53.295 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:53.295 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:53.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.313+0000 7f2965d9b640 1 -- 172.21.15.170:0/3976759396 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2960001af0 con 0x7f2968143d10 2024-03-29T11:15:53.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.313+0000 7f2965d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:53.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.313+0000 7f2965d9b640 1 -- 172.21.15.170:0/3976759396 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f2960001f40 con 0x7f2968143d10 2024-03-29T11:15:53.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.313+0000 7f2965d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:53.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.313+0000 7f2965d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:53.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.313+0000 7f2965d9b640 10 monclient: dump: 2024-03-29T11:15:53.296 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:53.296 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:53.296 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:53.297 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:53.297 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:53.297 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:53.297 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:53.297 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:53.297 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:53.297 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:53.299 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.313+0000 7f296d154640 5 monclient: authenticate success, global_id 4371 2024-03-29T11:15:53.299 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.313+0000 7f296d154640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:53.299 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.313+0000 7f296d154640 4 set_mon_vals no callback set 2024-03-29T11:15:53.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.314+0000 7f296d154640 10 monclient: shutdown 2024-03-29T11:15:53.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.314+0000 7f296d154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:53.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.314+0000 7f296d154640 1 -- 172.21.15.170:0/3976759396 >> v1:172.21.15.170:6789/0 conn(0x7f2968143d10 legacy=0x7f2968146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:53.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.317+0000 7f296d154640 1 -- 172.21.15.170:0/3976759396 shutdown_connections 2024-03-29T11:15:53.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.317+0000 7f296d154640 1 -- 172.21.15.170:0/3976759396 >> 172.21.15.170:0/3976759396 conn(0x7f2968066a00 msgr2=0x7f2968066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:53.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.318+0000 7f296d154640 1 -- 172.21.15.170:0/3976759396 shutdown_connections 2024-03-29T11:15:53.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.318+0000 7f296d154640 1 -- 172.21.15.170:0/3976759396 wait complete. 2024-03-29T11:15:53.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.319+0000 7f296d154640 10 monclient: build_initial_monmap 2024-03-29T11:15:53.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.319+0000 7f296d154640 10 monclient: monmap: 2024-03-29T11:15:53.300 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:15:53.300 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:53.300 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:15:53.300 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:15:53.300 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:53.300 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:53.301 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:15:53.301 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:15:53.301 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:15:53.301 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:53.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.320+0000 7f296d154640 1 librados: starting msgr at 2024-03-29T11:15:53.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.320+0000 7f296d154640 1 librados: starting objecter 2024-03-29T11:15:53.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.320+0000 7f296d154640 1 Processor -- start 2024-03-29T11:15:53.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.320+0000 7f296d154640 1 -- start start 2024-03-29T11:15:53.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.320+0000 7f296d154640 1 librados: setting wanted keys 2024-03-29T11:15:53.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.320+0000 7f296d154640 1 librados: calling monclient init 2024-03-29T11:15:53.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.320+0000 7f296d154640 10 monclient: init 2024-03-29T11:15:53.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.321+0000 7f296d154640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:53.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.321+0000 7f296d154640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:15:53.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.321+0000 7f296d154640 10 monclient(hunting): picked mon.noname-b con 0x7f29680b1f60 addr v1:172.21.15.184:6789/0 2024-03-29T11:15:53.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.321+0000 7f296d154640 10 monclient(hunting): picked mon.noname-c con 0x7f2968143d10 addr v1:172.21.15.170:6790/0 2024-03-29T11:15:53.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.321+0000 7f296d154640 10 monclient(hunting): picked mon.noname-a con 0x7f2968070280 addr v1:172.21.15.170:6789/0 2024-03-29T11:15:53.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.321+0000 7f296d154640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296806fdf0 con 0x7f2968070280 2024-03-29T11:15:53.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.322+0000 7f296d154640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29681394f0 con 0x7f29680b1f60 2024-03-29T11:15:53.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.322+0000 7f296d154640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29681397d0 con 0x7f2968143d10 2024-03-29T11:15:53.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.322+0000 7f296d154640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:53.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.322+0000 7f296d154640 10 monclient(hunting): authenticate will time out at 1134.634034s 2024-03-29T11:15:53.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.325+0000 7f296659c640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f2968143d10 0x7f296813ff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:34063/0 (socket says 172.21.15.170:34063) 2024-03-29T11:15:53.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.325+0000 7f296659c640 1 -- 172.21.15.170:0/3568823960 learned_addr learned my addr 172.21.15.170:0/3568823960 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:15:53.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.325+0000 7f294f7fe640 10 client.?.objecter ms_handle_connect 0x7f2968070280 2024-03-29T11:15:53.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.325+0000 7f294f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:53.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.325+0000 7f294f7fe640 10 client.?.objecter ms_handle_connect 0x7f2968143d10 2024-03-29T11:15:53.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.325+0000 7f294f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:53.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.326+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888539325 0 0) 0x7f29681397d0 con 0x7f2968143d10 2024-03-29T11:15:53.306 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.326+0000 7f294f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:53.306 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.326+0000 7f294f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:53.306 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.326+0000 7f294f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:53.306 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.326+0000 7f294f7fe640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:15:53.306 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.326+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2954004eb0 con 0x7f2968143d10 2024-03-29T11:15:53.306 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.326+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2705398864 0 0) 0x7f2954004eb0 con 0x7f2968143d10 2024-03-29T11:15:53.306 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.326+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29681397d0 con 0x7f2968143d10 2024-03-29T11:15:53.306 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.326+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f2958001890 con 0x7f2968143d10 2024-03-29T11:15:53.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.326+0000 7f294f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:53.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.326+0000 7f294f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:53.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.326+0000 7f294f7fe640 10 monclient(hunting): dump: 2024-03-29T11:15:53.307 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:53.307 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:53.307 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:53.307 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:53.307 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:53.307 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:53.307 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:53.307 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:53.307 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:53.307 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:53.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.326+0000 7f294f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:53.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.327+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042997857 0 0) 0x7f296806fdf0 con 0x7f2968070280 2024-03-29T11:15:53.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.327+0000 7f294f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:53.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.327+0000 7f294f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:53.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.327+0000 7f294f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:53.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.327+0000 7f294f7fe640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:15:53.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.327+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2954004eb0 con 0x7f2968070280 2024-03-29T11:15:53.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.329+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4102523897 0 0) 0x7f2954004eb0 con 0x7f2968070280 2024-03-29T11:15:53.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.329+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f296806fdf0 con 0x7f2968070280 2024-03-29T11:15:53.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.329+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f2960001e00 con 0x7f2968070280 2024-03-29T11:15:53.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.329+0000 7f294f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:53.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.329+0000 7f294f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:53.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.329+0000 7f294f7fe640 10 monclient(hunting): dump: 2024-03-29T11:15:53.309 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:53.309 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:53.309 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:53.309 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:53.309 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:53.309 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:53.309 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:53.310 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:53.310 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:53.310 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:53.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.330+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3025033134 0 0) 0x7f296806fdf0 con 0x7f2968070280 2024-03-29T11:15:53.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.330+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 >> v1:172.21.15.170:6790/0 conn(0x7f2968143d10 legacy=0x7f296813ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:53.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.330+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 >> v1:172.21.15.184:6789/0 conn(0x7f29680b1f60 legacy=0x7f2968141090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:53.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.330+0000 7f294f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:15:53.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.330+0000 7f294f7fe640 1 monclient: found mon.a 2024-03-29T11:15:53.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.330+0000 7f294f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:53.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.330+0000 7f294f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:15:53.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.330+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29681417a0 con 0x7f2968070280 2024-03-29T11:15:53.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.330+0000 7f294f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:15:53.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.330+0000 7f294f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:15:53.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.330+0000 7f294f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:53.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.330+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2960004b60 con 0x7f2968070280 2024-03-29T11:15:53.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.330+0000 7f294f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:53.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.330+0000 7f2964d99640 4 set_mon_vals no callback set 2024-03-29T11:15:53.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.330+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f2960004eb0 con 0x7f2968070280 2024-03-29T11:15:53.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.330+0000 7f294f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:53.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.330+0000 7f294f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:53.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.330+0000 7f294f7fe640 10 monclient: dump: 2024-03-29T11:15:53.311 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:15:53.311 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:15:53.311 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:53.311 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:15:53.311 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:53.311 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:15:53.312 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:15:53.312 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:15:53.312 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:15:53.312 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:15:53.312 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.331+0000 7f296d154640 5 monclient: authenticate success, global_id 4374 2024-03-29T11:15:53.312 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.331+0000 7f296d154640 10 monclient: _renew_subs 2024-03-29T11:15:53.312 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.331+0000 7f296d154640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:15:53.312 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.331+0000 7f296d154640 1 -- 172.21.15.170:0/3568823960 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2968137930 con 0x7f2968070280 2024-03-29T11:15:53.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.331+0000 7f296d154640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:53.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.331+0000 7f296d154640 10 monclient: _renew_subs 2024-03-29T11:15:53.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.331+0000 7f296d154640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:15:53.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.331+0000 7f296d154640 1 -- 172.21.15.170:0/3568823960 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f296806f640 con 0x7f2968070280 2024-03-29T11:15:53.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.331+0000 7f296d154640 1 librados: init done 2024-03-29T11:15:53.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.334+0000 7f296d154640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:53.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.334+0000 7f296d154640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:53.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.334+0000 7f296d154640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:15:53.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.334+0000 7f296d154640 1 -- 172.21.15.170:0/3568823960 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2968075f10 con 0x7f2968070280 2024-03-29T11:15:53.314 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.335+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f296001af00 con 0x7f2968070280 2024-03-29T11:15:53.315 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.336+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f2960084ce0 con 0x7f2968070280 2024-03-29T11:15:53.315 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.336+0000 7f294f7fe640 10 client.4374.objecter ms_dispatch 0x7f2968000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:53.315 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.336+0000 7f294f7fe640 3 client.4374.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:53.315 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.336+0000 7f294f7fe640 3 client.4374.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:53.315 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.336+0000 7f294f7fe640 20 client.4374.objecter dump_active .. 0 homeless 2024-03-29T11:15:53.318 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.340+0000 7f2966d9d640 10 monclient: get_auth_request con 0x7f295406def0 auth_method 0 2024-03-29T11:15:53.321 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.343+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f29600505c0 con 0x7f2968070280 2024-03-29T11:15:53.321 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.343+0000 7f294f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:53.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.343+0000 7f294f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:53.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.344+0000 7f294f7fe640 10 client.4374.objecter ms_handle_connect 0x7f295406def0 2024-03-29T11:15:53.644 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.666+0000 7f296d154640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:53.645 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.666+0000 7f296d154640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:53.645 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.666+0000 7f296d154640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:15:53.645 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.666+0000 7f296d154640 1 -- 172.21.15.170:0/3568823960 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f296806ff80 con 0x7f2968070280 2024-03-29T11:15:53.645 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.667+0000 7f294f7fe640 1 -- 172.21.15.170:0/3568823960 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f29600524f0 con 0x7f2968070280 2024-03-29T11:15:53.645 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.667+0000 7f294f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:53.645 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.667+0000 7f294f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:53.645 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:15:53.646 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:53.648 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.670+0000 7f296d154640 10 librados: watch_flush enter 2024-03-29T11:15:53.648 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.671+0000 7f296d154640 10 librados: watch_flush exit 2024-03-29T11:15:53.648 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.671+0000 7f296d154640 20 client.4374.objecter shutdown clearing up homeless session... 2024-03-29T11:15:53.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.671+0000 7f296d154640 10 client.4374.objecter successfully canceled tick 2024-03-29T11:15:53.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.671+0000 7f296d154640 1 -- 172.21.15.170:0/3568823960 >> v1:172.21.15.170:6816/3298994444 conn(0x7f295406def0 legacy=0x7f29540703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:53.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.671+0000 7f296d154640 10 monclient: shutdown 2024-03-29T11:15:53.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.671+0000 7f296d154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:53.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.671+0000 7f296d154640 1 -- 172.21.15.170:0/3568823960 >> v1:172.21.15.170:6789/0 conn(0x7f2968070280 legacy=0x7f29681e6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:53.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.671+0000 7f296d154640 1 -- 172.21.15.170:0/3568823960 shutdown_connections 2024-03-29T11:15:53.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.671+0000 7f296d154640 1 -- 172.21.15.170:0/3568823960 >> 172.21.15.170:0/3568823960 conn(0x7f2968066a00 msgr2=0x7f29680b7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:53.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.677+0000 7f296d154640 1 -- 172.21.15.170:0/3568823960 shutdown_connections 2024-03-29T11:15:53.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.678+0000 7f296d154640 1 -- 172.21.15.170:0/3568823960 wait complete. 2024-03-29T11:15:53.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:15:53.678+0000 7f296d154640 1 librados: shutdown 2024-03-29T11:15:53.674 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:00.482 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:00.638 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.657+0000 7fc51c372640 10 monclient: get_monmap_and_config 2024-03-29T11:16:00.638 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.657+0000 7fc51c372640 10 monclient: build_initial_monmap 2024-03-29T11:16:00.638 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.657+0000 7fc51c372640 10 monclient: monmap: 2024-03-29T11:16:00.638 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:16:00.638 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:00.639 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:16:00.659199+0000 2024-03-29T11:16:00.639 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:16:00.659199+0000 2024-03-29T11:16:00.639 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:00.639 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:00.639 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:16:00.639 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:16:00.639 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:16:00.639 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:00.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.659+0000 7fc51c372640 1 Processor -- start 2024-03-29T11:16:00.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.659+0000 7fc51c372640 1 -- start start 2024-03-29T11:16:00.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.659+0000 7fc51c372640 10 monclient: init 2024-03-29T11:16:00.640 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.660+0000 7fc51c372640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:00.640 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.660+0000 7fc51c372640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:16:00.640 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.660+0000 7fc51c372640 10 monclient(hunting): picked mon.noname-b con 0x7fc5140b1f60 addr v1:172.21.15.170:6790/0 2024-03-29T11:16:00.640 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.660+0000 7fc51c372640 10 monclient(hunting): picked mon.noname-a con 0x7fc514071eb0 addr v1:172.21.15.170:6789/0 2024-03-29T11:16:00.640 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.660+0000 7fc51c372640 10 monclient(hunting): picked mon.noname-c con 0x7fc514143d10 addr v1:172.21.15.184:6789/0 2024-03-29T11:16:00.640 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.660+0000 7fc51c372640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc51406ef80 con 0x7fc514071eb0 2024-03-29T11:16:00.640 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.660+0000 7fc51c372640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc51406f260 con 0x7fc514143d10 2024-03-29T11:16:00.640 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.660+0000 7fc51c372640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5140b40c0 con 0x7fc5140b1f60 2024-03-29T11:16:00.640 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.660+0000 7fc51c372640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:00.640 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.660+0000 7fc51c372640 10 monclient(hunting): authenticate will time out at 1141.972465s 2024-03-29T11:16:00.642 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.664+0000 7fc51990d640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fc514071eb0 0x7fc514146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:53715/0 (socket says 172.21.15.170:53715) 2024-03-29T11:16:00.642 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.664+0000 7fc51990d640 1 -- 172.21.15.170:0/3217912271 learned_addr learned my addr 172.21.15.170:0/3217912271 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:16:00.642 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.664+0000 7fc51910c640 1 -- 172.21.15.170:0/3217912271 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703261265 0 0) 0x7fc51406ef80 con 0x7fc514071eb0 2024-03-29T11:16:00.642 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.664+0000 7fc51910c640 10 monclient(hunting): _negotiate 2024-03-29T11:16:00.642 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.664+0000 7fc51910c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:00.643 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51910c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:00.643 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51910c640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:16:00.643 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51910c640 1 -- 172.21.15.170:0/3217912271 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc500003520 con 0x7fc514071eb0 2024-03-29T11:16:00.643 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51910c640 1 -- 172.21.15.170:0/3217912271 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3524286536 0 0) 0x7fc500003520 con 0x7fc514071eb0 2024-03-29T11:16:00.644 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51910c640 1 -- 172.21.15.170:0/3217912271 >> v1:172.21.15.170:6790/0 conn(0x7fc5140b1f60 legacy=0x7fc514143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:00.644 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51910c640 1 -- 172.21.15.170:0/3217912271 >> v1:172.21.15.184:6789/0 conn(0x7fc514143d10 legacy=0x7fc514146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:00.644 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51910c640 10 monclient: _finish_hunting 0 2024-03-29T11:16:00.644 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51910c640 1 monclient: found mon.noname-a 2024-03-29T11:16:00.644 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51910c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:00.644 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51910c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:16:00.644 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51910c640 1 -- 172.21.15.170:0/3217912271 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc514155d70 con 0x7fc514071eb0 2024-03-29T11:16:00.644 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51910c640 10 monclient: _finish_auth 0 2024-03-29T11:16:00.644 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51910c640 10 monclient: _check_auth_tickets 2024-03-29T11:16:00.645 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51910c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:00.645 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51c372640 5 monclient: authenticate success, global_id 4377 2024-03-29T11:16:00.645 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51c372640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:00.645 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51910c640 1 -- 172.21.15.170:0/3217912271 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fc504001690 con 0x7fc514071eb0 2024-03-29T11:16:00.645 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51910c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:00.645 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51910c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:00.645 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51910c640 10 monclient: dump: 2024-03-29T11:16:00.645 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:00.645 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:00.645 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:00.645 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:00.645 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:00.645 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:00.645 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:00.646 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:00.646 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:00.646 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:00.646 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.665+0000 7fc51c372640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:00.646 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.666+0000 7fc51910c640 1 -- 172.21.15.170:0/3217912271 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc504001920 con 0x7fc514071eb0 2024-03-29T11:16:00.646 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.666+0000 7fc51910c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:00.646 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.666+0000 7fc51c372640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:00.646 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.666+0000 7fc51c372640 4 set_mon_vals no callback set 2024-03-29T11:16:00.646 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.667+0000 7fc51c372640 10 monclient: shutdown 2024-03-29T11:16:00.647 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.667+0000 7fc51c372640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:00.647 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.667+0000 7fc51c372640 1 -- 172.21.15.170:0/3217912271 >> v1:172.21.15.170:6789/0 conn(0x7fc514071eb0 legacy=0x7fc514146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:00.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.671+0000 7fc51c372640 1 -- 172.21.15.170:0/3217912271 shutdown_connections 2024-03-29T11:16:00.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.671+0000 7fc51c372640 1 -- 172.21.15.170:0/3217912271 >> 172.21.15.170:0/3217912271 conn(0x7fc514066a00 msgr2=0x7fc514066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:00.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.674+0000 7fc51c372640 1 -- 172.21.15.170:0/3217912271 shutdown_connections 2024-03-29T11:16:00.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.676+0000 7fc51c372640 1 -- 172.21.15.170:0/3217912271 wait complete. 2024-03-29T11:16:00.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.677+0000 7fc51c372640 10 monclient: build_initial_monmap 2024-03-29T11:16:00.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.677+0000 7fc51c372640 10 monclient: monmap: 2024-03-29T11:16:00.655 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:16:00.656 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:00.656 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:16:00.656 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:16:00.656 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:00.656 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:00.656 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:16:00.656 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:16:00.656 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:16:00.656 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:00.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.677+0000 7fc51c372640 1 librados: starting msgr at 2024-03-29T11:16:00.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.677+0000 7fc51c372640 1 librados: starting objecter 2024-03-29T11:16:00.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.677+0000 7fc51c372640 1 Processor -- start 2024-03-29T11:16:00.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.678+0000 7fc51c372640 1 -- start start 2024-03-29T11:16:00.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.678+0000 7fc51c372640 1 librados: setting wanted keys 2024-03-29T11:16:00.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.678+0000 7fc51c372640 1 librados: calling monclient init 2024-03-29T11:16:00.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.678+0000 7fc51c372640 10 monclient: init 2024-03-29T11:16:00.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.679+0000 7fc51c372640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:00.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.679+0000 7fc51c372640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:00.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.679+0000 7fc51c372640 10 monclient(hunting): picked mon.noname-a con 0x7fc5140b1f60 addr v1:172.21.15.170:6789/0 2024-03-29T11:16:00.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.679+0000 7fc51c372640 10 monclient(hunting): picked mon.noname-b con 0x7fc514143d10 addr v1:172.21.15.184:6789/0 2024-03-29T11:16:00.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.679+0000 7fc51c372640 10 monclient(hunting): picked mon.noname-c con 0x7fc5141354b0 addr v1:172.21.15.170:6790/0 2024-03-29T11:16:00.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.679+0000 7fc51c372640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5141393f0 con 0x7fc5140b1f60 2024-03-29T11:16:00.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.679+0000 7fc51c372640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc514154a70 con 0x7fc514143d10 2024-03-29T11:16:00.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.679+0000 7fc51c372640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc514139640 con 0x7fc5141354b0 2024-03-29T11:16:00.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.679+0000 7fc51c372640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:00.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.679+0000 7fc51c372640 10 monclient(hunting): authenticate will time out at 1141.991230s 2024-03-29T11:16:00.661 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.682+0000 7fc51a10e640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fc5140b1f60 0x7fc514154360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:53867/0 (socket says 172.21.15.170:53867) 2024-03-29T11:16:00.661 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.682+0000 7fc51a10e640 1 -- 172.21.15.170:0/2222793374 learned_addr learned my addr 172.21.15.170:0/2222793374 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:16:00.661 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.683+0000 7fc50affd640 10 client.?.objecter ms_handle_connect 0x7fc5140b1f60 2024-03-29T11:16:00.661 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.683+0000 7fc50affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:00.662 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.683+0000 7fc50affd640 1 -- 172.21.15.170:0/2222793374 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845961583 0 0) 0x7fc5141393f0 con 0x7fc5140b1f60 2024-03-29T11:16:00.662 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.683+0000 7fc50affd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:00.662 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.683+0000 7fc50affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:00.662 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.683+0000 7fc50affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:00.662 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.683+0000 7fc50affd640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:16:00.662 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.683+0000 7fc50affd640 1 -- 172.21.15.170:0/2222793374 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4f8004eb0 con 0x7fc5140b1f60 2024-03-29T11:16:00.662 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.684+0000 7fc50affd640 1 -- 172.21.15.170:0/2222793374 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2425084153 0 0) 0x7fc4f8004eb0 con 0x7fc5140b1f60 2024-03-29T11:16:00.663 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.684+0000 7fc50affd640 1 -- 172.21.15.170:0/2222793374 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5141393f0 con 0x7fc5140b1f60 2024-03-29T11:16:00.663 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.684+0000 7fc50affd640 1 -- 172.21.15.170:0/2222793374 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fc510001a30 con 0x7fc5140b1f60 2024-03-29T11:16:00.663 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.686+0000 7fc50affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:00.664 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.686+0000 7fc50affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:00.664 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.686+0000 7fc50affd640 10 monclient(hunting): dump: 2024-03-29T11:16:00.664 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:00.664 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:00.664 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:00.664 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:00.664 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:00.664 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:00.664 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:00.664 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:00.664 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:00.664 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:00.664 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.686+0000 7fc50affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:00.665 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.686+0000 7fc50affd640 10 client.?.objecter ms_handle_connect 0x7fc514143d10 2024-03-29T11:16:00.665 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.686+0000 7fc50affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:00.667 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.688+0000 7fc50affd640 1 -- 172.21.15.170:0/2222793374 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1870410169 0 0) 0x7fc5141393f0 con 0x7fc5140b1f60 2024-03-29T11:16:00.667 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.688+0000 7fc50affd640 1 -- 172.21.15.170:0/2222793374 >> v1:172.21.15.170:6790/0 conn(0x7fc5141354b0 legacy=0x7fc5141e7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:00.667 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.688+0000 7fc50affd640 1 -- 172.21.15.170:0/2222793374 >> v1:172.21.15.184:6789/0 conn(0x7fc514143d10 legacy=0x7fc514134da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:00.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.688+0000 7fc50affd640 10 monclient: _finish_hunting 0 2024-03-29T11:16:00.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.688+0000 7fc50affd640 1 monclient: found mon.a 2024-03-29T11:16:00.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.688+0000 7fc50affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:00.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.688+0000 7fc50affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:00.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.689+0000 7fc50affd640 1 -- 172.21.15.170:0/2222793374 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc514153e40 con 0x7fc5140b1f60 2024-03-29T11:16:00.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.689+0000 7fc50affd640 10 monclient: _finish_auth 0 2024-03-29T11:16:00.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.689+0000 7fc50affd640 10 monclient: _check_auth_tickets 2024-03-29T11:16:00.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.689+0000 7fc50affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:00.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.689+0000 7fc50affd640 1 -- 172.21.15.170:0/2222793374 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc510001c90 con 0x7fc5140b1f60 2024-03-29T11:16:00.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.689+0000 7fc50affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:00.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.689+0000 7fc51890b640 4 set_mon_vals no callback set 2024-03-29T11:16:00.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.689+0000 7fc50affd640 1 -- 172.21.15.170:0/2222793374 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fc510003aa0 con 0x7fc5140b1f60 2024-03-29T11:16:00.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.689+0000 7fc50affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:00.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.689+0000 7fc50affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:00.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.689+0000 7fc50affd640 10 monclient: dump: 2024-03-29T11:16:00.669 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:00.669 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:00.669 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:00.669 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:00.669 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:00.669 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:00.669 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:00.669 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:00.669 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:00.669 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:00.670 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.690+0000 7fc51c372640 5 monclient: authenticate success, global_id 4380 2024-03-29T11:16:00.670 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.690+0000 7fc51c372640 10 monclient: _renew_subs 2024-03-29T11:16:00.670 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.690+0000 7fc51c372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:00.670 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.690+0000 7fc51c372640 1 -- 172.21.15.170:0/2222793374 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc514142bd0 con 0x7fc5140b1f60 2024-03-29T11:16:00.670 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.690+0000 7fc51c372640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:00.670 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.690+0000 7fc51c372640 10 monclient: _renew_subs 2024-03-29T11:16:00.670 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.690+0000 7fc51c372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:00.670 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.690+0000 7fc51c372640 1 -- 172.21.15.170:0/2222793374 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5141ded30 con 0x7fc5140b1f60 2024-03-29T11:16:00.670 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.690+0000 7fc51c372640 1 librados: init done 2024-03-29T11:16:00.670 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.691+0000 7fc51c372640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:00.670 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.691+0000 7fc51c372640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:00.671 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.691+0000 7fc51c372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:00.671 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.691+0000 7fc51c372640 1 -- 172.21.15.170:0/2222793374 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc514153ca0 con 0x7fc5140b1f60 2024-03-29T11:16:00.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.696+0000 7fc50affd640 1 -- 172.21.15.170:0/2222793374 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fc510003c00 con 0x7fc5140b1f60 2024-03-29T11:16:00.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.696+0000 7fc50affd640 1 -- 172.21.15.170:0/2222793374 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fc510084d10 con 0x7fc5140b1f60 2024-03-29T11:16:00.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.696+0000 7fc50affd640 10 client.4380.objecter ms_dispatch 0x7fc514000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:00.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.696+0000 7fc50affd640 3 client.4380.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:00.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.696+0000 7fc50affd640 3 client.4380.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:00.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.696+0000 7fc50affd640 20 client.4380.objecter dump_active .. 0 homeless 2024-03-29T11:16:00.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.698+0000 7fc51990d640 10 monclient: get_auth_request con 0x7fc4f806dd40 auth_method 0 2024-03-29T11:16:00.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.698+0000 7fc50affd640 10 client.4380.objecter ms_handle_connect 0x7fc4f806dd40 2024-03-29T11:16:00.679 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.701+0000 7fc50affd640 1 -- 172.21.15.170:0/2222793374 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc514153ca0 con 0x7fc5140b1f60 2024-03-29T11:16:00.679 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.701+0000 7fc50affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:00.679 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:00.701+0000 7fc50affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:01.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.032+0000 7fc51c372640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:01.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.032+0000 7fc51c372640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:01.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.032+0000 7fc51c372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:01.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.032+0000 7fc51c372640 1 -- 172.21.15.170:0/2222793374 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc514153ca0 con 0x7fc5140b1f60 2024-03-29T11:16:01.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.033+0000 7fc50affd640 1 -- 172.21.15.170:0/2222793374 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fc514153ca0 con 0x7fc5140b1f60 2024-03-29T11:16:01.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.033+0000 7fc50affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:01.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.033+0000 7fc50affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:01.011 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:16:01.011 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:01.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.036+0000 7fc51c372640 10 librados: watch_flush enter 2024-03-29T11:16:01.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.036+0000 7fc51c372640 10 librados: watch_flush exit 2024-03-29T11:16:01.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.036+0000 7fc51c372640 20 client.4380.objecter shutdown clearing up homeless session... 2024-03-29T11:16:01.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.036+0000 7fc51c372640 10 client.4380.objecter successfully canceled tick 2024-03-29T11:16:01.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.037+0000 7fc51c372640 1 -- 172.21.15.170:0/2222793374 >> v1:172.21.15.170:6816/3298994444 conn(0x7fc4f806dd40 legacy=0x7fc4f8070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:01.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.037+0000 7fc51c372640 10 monclient: shutdown 2024-03-29T11:16:01.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.037+0000 7fc51c372640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:01.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.037+0000 7fc51c372640 1 -- 172.21.15.170:0/2222793374 >> v1:172.21.15.170:6789/0 conn(0x7fc5140b1f60 legacy=0x7fc514154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:01.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.037+0000 7fc51c372640 1 -- 172.21.15.170:0/2222793374 shutdown_connections 2024-03-29T11:16:01.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.037+0000 7fc51c372640 1 -- 172.21.15.170:0/2222793374 >> 172.21.15.170:0/2222793374 conn(0x7fc514066a00 msgr2=0x7fc514066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:01.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.038+0000 7fc51c372640 1 -- 172.21.15.170:0/2222793374 shutdown_connections 2024-03-29T11:16:01.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.039+0000 7fc51c372640 1 -- 172.21.15.170:0/2222793374 wait complete. 2024-03-29T11:16:01.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:01.039+0000 7fc51c372640 1 librados: shutdown 2024-03-29T11:16:01.034 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:07.842 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:07.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.018+0000 7f6e2cb25640 10 monclient: get_monmap_and_config 2024-03-29T11:16:07.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.018+0000 7f6e2cb25640 10 monclient: build_initial_monmap 2024-03-29T11:16:07.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.018+0000 7f6e2cb25640 10 monclient: monmap: 2024-03-29T11:16:07.995 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:16:07.995 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:07.995 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:16:08.018778+0000 2024-03-29T11:16:07.995 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:16:08.018778+0000 2024-03-29T11:16:07.995 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:07.995 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:07.995 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:16:07.996 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:16:07.996 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:16:07.996 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:07.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.021+0000 7f6e2cb25640 1 Processor -- start 2024-03-29T11:16:07.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.021+0000 7f6e2cb25640 1 -- start start 2024-03-29T11:16:07.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.021+0000 7f6e2cb25640 10 monclient: init 2024-03-29T11:16:07.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.022+0000 7f6e2cb25640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:07.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.022+0000 7f6e2cb25640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:07.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.022+0000 7f6e2cb25640 10 monclient(hunting): picked mon.noname-c con 0x7f6e281457f0 addr v1:172.21.15.184:6789/0 2024-03-29T11:16:07.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.022+0000 7f6e2cb25640 10 monclient(hunting): picked mon.noname-a con 0x7f6e2813f650 addr v1:172.21.15.170:6789/0 2024-03-29T11:16:07.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.022+0000 7f6e2cb25640 10 monclient(hunting): picked mon.noname-b con 0x7f6e280a9d70 addr v1:172.21.15.170:6790/0 2024-03-29T11:16:07.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.022+0000 7f6e2cb25640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e280aaef0 con 0x7f6e2813f650 2024-03-29T11:16:07.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.022+0000 7f6e2cb25640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e28145bd0 con 0x7f6e281457f0 2024-03-29T11:16:07.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.022+0000 7f6e2cb25640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e28145eb0 con 0x7f6e280a9d70 2024-03-29T11:16:07.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.022+0000 7f6e2cb25640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:08.000 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.022+0000 7f6e2cb25640 10 monclient(hunting): authenticate will time out at 1149.333637s 2024-03-29T11:16:08.004 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.027+0000 7f6e25d9b640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f6e2813f650 0x7f6e280b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:38627/0 (socket says 172.21.15.170:38627) 2024-03-29T11:16:08.004 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.027+0000 7f6e25d9b640 1 -- 172.21.15.170:0/823904937 learned_addr learned my addr 172.21.15.170:0/823904937 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:16:08.005 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.028+0000 7f6e2559a640 1 -- 172.21.15.170:0/823904937 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100308377 0 0) 0x7f6e280aaef0 con 0x7f6e2813f650 2024-03-29T11:16:08.005 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.028+0000 7f6e2559a640 10 monclient(hunting): _negotiate 2024-03-29T11:16:08.005 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.028+0000 7f6e2559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:08.005 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.028+0000 7f6e2559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:08.005 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.028+0000 7f6e2559a640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:16:08.005 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.028+0000 7f6e2559a640 1 -- 172.21.15.170:0/823904937 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e10003520 con 0x7f6e2813f650 2024-03-29T11:16:08.006 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 1 -- 172.21.15.170:0/823904937 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 548149293 0 0) 0x7f6e10003520 con 0x7f6e2813f650 2024-03-29T11:16:08.006 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 1 -- 172.21.15.170:0/823904937 >> v1:172.21.15.170:6790/0 conn(0x7f6e280a9d70 legacy=0x7f6e2814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:08.006 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 1 -- 172.21.15.170:0/823904937 >> v1:172.21.15.184:6789/0 conn(0x7f6e281457f0 legacy=0x7f6e2813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:08.006 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 10 monclient: _finish_hunting 0 2024-03-29T11:16:08.006 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 1 monclient: found mon.noname-a 2024-03-29T11:16:08.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:08.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:16:08.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 1 -- 172.21.15.170:0/823904937 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e28146340 con 0x7f6e2813f650 2024-03-29T11:16:08.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 10 monclient: _finish_auth 0 2024-03-29T11:16:08.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 10 monclient: _check_auth_tickets 2024-03-29T11:16:08.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:08.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 1 -- 172.21.15.170:0/823904937 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f6e1c001690 con 0x7f6e2813f650 2024-03-29T11:16:08.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:08.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:08.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 10 monclient: dump: 2024-03-29T11:16:08.007 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:08.007 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:08.007 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:08.007 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:08.007 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:08.008 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:08.008 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:08.008 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:08.008 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:08.008 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:08.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 1 -- 172.21.15.170:0/823904937 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e1c001920 con 0x7f6e2813f650 2024-03-29T11:16:08.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:08.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 1 -- 172.21.15.170:0/823904937 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f6e1c001cb0 con 0x7f6e2813f650 2024-03-29T11:16:08.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:08.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:08.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2559a640 10 monclient: dump: 2024-03-29T11:16:08.008 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:08.008 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:08.008 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:08.009 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:08.009 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:08.009 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:08.009 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:08.009 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:08.009 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:08.009 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:08.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2cb25640 5 monclient: authenticate success, global_id 4383 2024-03-29T11:16:08.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2cb25640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:08.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.029+0000 7f6e2cb25640 4 set_mon_vals no callback set 2024-03-29T11:16:08.013 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.030+0000 7f6e2cb25640 10 monclient: shutdown 2024-03-29T11:16:08.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.030+0000 7f6e2cb25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:08.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.030+0000 7f6e2cb25640 1 -- 172.21.15.170:0/823904937 >> v1:172.21.15.170:6789/0 conn(0x7f6e2813f650 legacy=0x7f6e280b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:08.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.030+0000 7f6e2cb25640 1 -- 172.21.15.170:0/823904937 shutdown_connections 2024-03-29T11:16:08.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.030+0000 7f6e2cb25640 1 -- 172.21.15.170:0/823904937 >> 172.21.15.170:0/823904937 conn(0x7f6e28066a00 msgr2=0x7f6e28066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:08.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.030+0000 7f6e2cb25640 1 -- 172.21.15.170:0/823904937 shutdown_connections 2024-03-29T11:16:08.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.030+0000 7f6e2cb25640 1 -- 172.21.15.170:0/823904937 wait complete. 2024-03-29T11:16:08.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.031+0000 7f6e2cb25640 10 monclient: build_initial_monmap 2024-03-29T11:16:08.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.031+0000 7f6e2cb25640 10 monclient: monmap: 2024-03-29T11:16:08.014 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:16:08.014 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:08.014 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:16:08.014 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:16:08.014 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:08.015 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:08.015 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:16:08.015 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:16:08.015 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:16:08.015 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:08.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.032+0000 7f6e2cb25640 1 librados: starting msgr at 2024-03-29T11:16:08.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.032+0000 7f6e2cb25640 1 librados: starting objecter 2024-03-29T11:16:08.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.032+0000 7f6e2cb25640 1 Processor -- start 2024-03-29T11:16:08.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.032+0000 7f6e2cb25640 1 -- start start 2024-03-29T11:16:08.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.032+0000 7f6e2cb25640 1 librados: setting wanted keys 2024-03-29T11:16:08.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.032+0000 7f6e2cb25640 1 librados: calling monclient init 2024-03-29T11:16:08.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.032+0000 7f6e2cb25640 10 monclient: init 2024-03-29T11:16:08.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.033+0000 7f6e2cb25640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:08.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.033+0000 7f6e2cb25640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:16:08.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.033+0000 7f6e2cb25640 10 monclient(hunting): picked mon.noname-c con 0x7f6e280a9d70 addr v1:172.21.15.170:6790/0 2024-03-29T11:16:08.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.033+0000 7f6e2cb25640 10 monclient(hunting): picked mon.noname-a con 0x7f6e2813f650 addr v1:172.21.15.170:6789/0 2024-03-29T11:16:08.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.033+0000 7f6e2cb25640 10 monclient(hunting): picked mon.noname-b con 0x7f6e281457f0 addr v1:172.21.15.184:6789/0 2024-03-29T11:16:08.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.033+0000 7f6e2cb25640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e281e1bb0 con 0x7f6e2813f650 2024-03-29T11:16:08.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.033+0000 7f6e2cb25640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e28142bf0 con 0x7f6e281457f0 2024-03-29T11:16:08.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.033+0000 7f6e2cb25640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e28141e90 con 0x7f6e280a9d70 2024-03-29T11:16:08.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.033+0000 7f6e2cb25640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:08.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.033+0000 7f6e2cb25640 10 monclient(hunting): authenticate will time out at 1149.344625s 2024-03-29T11:16:08.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.038+0000 7f6e26d9d640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f6e281457f0 0x7f6e281e3fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:58937/0 (socket says 172.21.15.170:58937) 2024-03-29T11:16:08.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.038+0000 7f6e26d9d640 1 -- 172.21.15.170:0/146683903 learned_addr learned my addr 172.21.15.170:0/146683903 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:16:08.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.039+0000 7f6e16ffd640 10 client.?.objecter ms_handle_connect 0x7f6e281457f0 2024-03-29T11:16:08.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.039+0000 7f6e16ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:08.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.040+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472950976 0 0) 0x7f6e28142bf0 con 0x7f6e281457f0 2024-03-29T11:16:08.018 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.040+0000 7f6e16ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:08.018 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.040+0000 7f6e16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:08.018 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.040+0000 7f6e16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:08.018 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.040+0000 7f6e16ffd640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:16:08.018 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.040+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e08004eb0 con 0x7f6e281457f0 2024-03-29T11:16:08.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.043+0000 7f6e16ffd640 10 client.?.objecter ms_handle_connect 0x7f6e280a9d70 2024-03-29T11:16:08.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:08.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459601479 0 0) 0x7f6e28141e90 con 0x7f6e280a9d70 2024-03-29T11:16:08.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:08.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:08.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:08.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:16:08.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e28142bf0 con 0x7f6e280a9d70 2024-03-29T11:16:08.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 10 client.?.objecter ms_handle_connect 0x7f6e2813f650 2024-03-29T11:16:08.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:08.022 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3234178459 0 0) 0x7f6e08004eb0 con 0x7f6e281457f0 2024-03-29T11:16:08.022 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e28141e90 con 0x7f6e281457f0 2024-03-29T11:16:08.022 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2367624937 0 0) 0x7f6e28142bf0 con 0x7f6e280a9d70 2024-03-29T11:16:08.022 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e08004eb0 con 0x7f6e280a9d70 2024-03-29T11:16:08.022 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f6e180019b0 con 0x7f6e281457f0 2024-03-29T11:16:08.022 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:08.022 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:08.022 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 10 monclient(hunting): dump: 2024-03-29T11:16:08.022 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:08.022 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:08.022 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:08.023 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:08.023 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:08.023 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:08.023 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:08.023 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:08.023 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:08.023 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:08.023 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:08.023 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f6e20001a90 con 0x7f6e280a9d70 2024-03-29T11:16:08.023 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:08.023 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:08.023 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 10 monclient(hunting): dump: 2024-03-29T11:16:08.023 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:08.023 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:08.023 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:08.024 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:08.024 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:08.024 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:08.024 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:08.024 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:08.024 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:08.024 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:08.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3694637767 0 0) 0x7f6e08004eb0 con 0x7f6e280a9d70 2024-03-29T11:16:08.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 >> v1:172.21.15.184:6789/0 conn(0x7f6e281457f0 legacy=0x7f6e281e3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:08.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.044+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 >> v1:172.21.15.170:6789/0 conn(0x7f6e2813f650 legacy=0x7f6e281e14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:08.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e16ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:16:08.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e16ffd640 1 monclient: found mon.c 2024-03-29T11:16:08.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:08.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e16ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:16:08.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e281420b0 con 0x7f6e280a9d70 2024-03-29T11:16:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e16ffd640 10 monclient: _finish_auth 0 2024-03-29T11:16:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e16ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:16:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e2cb25640 5 monclient: authenticate success, global_id 4403 2024-03-29T11:16:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e2cb25640 10 monclient: _renew_subs 2024-03-29T11:16:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e2cb25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:16:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e2cb25640 1 -- 172.21.15.170:0/146683903 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e28142400 con 0x7f6e280a9d70 2024-03-29T11:16:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e2cb25640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e2cb25640 10 monclient: _renew_subs 2024-03-29T11:16:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e2cb25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:16:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e2cb25640 1 -- 172.21.15.170:0/146683903 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e281e46c0 con 0x7f6e280a9d70 2024-03-29T11:16:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e2cb25640 1 librados: init done 2024-03-29T11:16:08.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e20001bf0 con 0x7f6e280a9d70 2024-03-29T11:16:08.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:08.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f6e20003b10 con 0x7f6e280a9d70 2024-03-29T11:16:08.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.045+0000 7f6e17fff640 4 set_mon_vals no callback set 2024-03-29T11:16:08.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.046+0000 7f6e2cb25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:08.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.046+0000 7f6e16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:08.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.046+0000 7f6e16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:08.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.046+0000 7f6e16ffd640 10 monclient: dump: 2024-03-29T11:16:08.028 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:08.028 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:08.028 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:08.028 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:08.028 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:08.028 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:08.028 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:08.028 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:08.028 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:08.028 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:08.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.047+0000 7f6e2cb25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.047+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f6e20003c70 con 0x7f6e280a9d70 2024-03-29T11:16:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.048+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f6e20083ac0 con 0x7f6e280a9d70 2024-03-29T11:16:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.048+0000 7f6e16ffd640 10 client.4403.objecter ms_dispatch 0x7f6e28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.048+0000 7f6e16ffd640 3 client.4403.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.048+0000 7f6e16ffd640 3 client.4403.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.048+0000 7f6e16ffd640 20 client.4403.objecter dump_active .. 0 homeless 2024-03-29T11:16:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.048+0000 7f6e2cb25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:16:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.048+0000 7f6e2cb25640 1 -- 172.21.15.170:0/146683903 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0c005080 con 0x7f6e280a9d70 2024-03-29T11:16:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.048+0000 7f6e25d9b640 10 monclient: get_auth_request con 0x7f6e0806de80 auth_method 0 2024-03-29T11:16:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.049+0000 7f6e16ffd640 10 client.4403.objecter ms_handle_connect 0x7f6e0806de80 2024-03-29T11:16:08.036 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.059+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e2004f3a0 con 0x7f6e280a9d70 2024-03-29T11:16:08.036 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.059+0000 7f6e16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:08.036 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.059+0000 7f6e16ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:08.364 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.387+0000 7f6e2cb25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:08.364 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.387+0000 7f6e2cb25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:08.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.387+0000 7f6e2cb25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:16:08.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.387+0000 7f6e2cb25640 1 -- 172.21.15.170:0/146683903 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e0c005330 con 0x7f6e280a9d70 2024-03-29T11:16:08.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.388+0000 7f6e16ffd640 1 -- 172.21.15.170:0/146683903 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f6e200512d0 con 0x7f6e280a9d70 2024-03-29T11:16:08.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.388+0000 7f6e16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:08.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.388+0000 7f6e16ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:08.366 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:16:08.367 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:08.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.392+0000 7f6e2cb25640 10 librados: watch_flush enter 2024-03-29T11:16:08.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.392+0000 7f6e2cb25640 10 librados: watch_flush exit 2024-03-29T11:16:08.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.392+0000 7f6e2cb25640 20 client.4403.objecter shutdown clearing up homeless session... 2024-03-29T11:16:08.369 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.392+0000 7f6e2cb25640 10 client.4403.objecter successfully canceled tick 2024-03-29T11:16:08.373 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.396+0000 7f6e2cb25640 1 -- 172.21.15.170:0/146683903 >> v1:172.21.15.170:6816/3298994444 conn(0x7f6e0806de80 legacy=0x7f6e08070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:08.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.396+0000 7f6e2cb25640 10 monclient: shutdown 2024-03-29T11:16:08.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.396+0000 7f6e2cb25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:08.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.396+0000 7f6e2cb25640 1 -- 172.21.15.170:0/146683903 >> v1:172.21.15.170:6790/0 conn(0x7f6e280a9d70 legacy=0x7f6e28144540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:08.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.396+0000 7f6e2cb25640 1 -- 172.21.15.170:0/146683903 shutdown_connections 2024-03-29T11:16:08.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.396+0000 7f6e2cb25640 1 -- 172.21.15.170:0/146683903 >> 172.21.15.170:0/146683903 conn(0x7f6e28066a00 msgr2=0x7f6e280b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:08.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.397+0000 7f6e2cb25640 1 -- 172.21.15.170:0/146683903 shutdown_connections 2024-03-29T11:16:08.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.397+0000 7f6e2cb25640 1 -- 172.21.15.170:0/146683903 wait complete. 2024-03-29T11:16:08.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:08.397+0000 7f6e2cb25640 1 librados: shutdown 2024-03-29T11:16:08.395 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:15.202 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:15.363 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.386+0000 7f61b0a29640 10 monclient: get_monmap_and_config 2024-03-29T11:16:15.363 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.386+0000 7f61b0a29640 10 monclient: build_initial_monmap 2024-03-29T11:16:15.363 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.386+0000 7f61b0a29640 10 monclient: monmap: 2024-03-29T11:16:15.363 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:16:15.364 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:15.364 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:16:15.386813+0000 2024-03-29T11:16:15.364 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:16:15.386813+0000 2024-03-29T11:16:15.364 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:15.364 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:15.364 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:16:15.364 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:16:15.364 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:16:15.364 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:15.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.387+0000 7f61b0a29640 1 Processor -- start 2024-03-29T11:16:15.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.387+0000 7f61b0a29640 1 -- start start 2024-03-29T11:16:15.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.387+0000 7f61b0a29640 10 monclient: init 2024-03-29T11:16:15.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.388+0000 7f61b0a29640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:15.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.388+0000 7f61b0a29640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:15.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.388+0000 7f61b0a29640 10 monclient(hunting): picked mon.noname-c con 0x7f61ac0a9750 addr v1:172.21.15.184:6789/0 2024-03-29T11:16:15.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.388+0000 7f61b0a29640 10 monclient(hunting): picked mon.noname-a con 0x7f61ac0add40 addr v1:172.21.15.170:6789/0 2024-03-29T11:16:15.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.388+0000 7f61b0a29640 10 monclient(hunting): picked mon.noname-b con 0x7f61ac0aad20 addr v1:172.21.15.170:6790/0 2024-03-29T11:16:15.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.388+0000 7f61b0a29640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ac149870 con 0x7f61ac0add40 2024-03-29T11:16:15.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.388+0000 7f61b0a29640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ac0abf80 con 0x7f61ac0a9750 2024-03-29T11:16:15.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.388+0000 7f61b0a29640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ac149ac0 con 0x7f61ac0aad20 2024-03-29T11:16:15.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.388+0000 7f61b0a29640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:15.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.388+0000 7f61b0a29640 10 monclient(hunting): authenticate will time out at 1156.699934s 2024-03-29T11:16:15.373 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.395+0000 7f61aad9d640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f61ac0a9750 0x7f61ac149fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:34775/0 (socket says 172.21.15.170:34775) 2024-03-29T11:16:15.373 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.395+0000 7f61aad9d640 1 -- 172.21.15.170:0/3500423698 learned_addr learned my addr 172.21.15.170:0/3500423698 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:16:15.373 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.395+0000 7f61a9d9b640 1 -- 172.21.15.170:0/3500423698 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627550890 0 0) 0x7f61ac0abf80 con 0x7f61ac0a9750 2024-03-29T11:16:15.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.395+0000 7f61a9d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:16:15.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.395+0000 7f61a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:15.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.395+0000 7f61a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:15.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.395+0000 7f61a9d9b640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:16:15.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.395+0000 7f61a9d9b640 1 -- 172.21.15.170:0/3500423698 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6194003520 con 0x7f61ac0a9750 2024-03-29T11:16:15.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.395+0000 7f61a9d9b640 1 -- 172.21.15.170:0/3500423698 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1445570196 0 0) 0x7f6194003520 con 0x7f61ac0a9750 2024-03-29T11:16:15.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.395+0000 7f61a9d9b640 1 -- 172.21.15.170:0/3500423698 >> v1:172.21.15.170:6790/0 conn(0x7f61ac0aad20 legacy=0x7f61ac151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:15.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.395+0000 7f61a9d9b640 1 -- 172.21.15.170:0/3500423698 >> v1:172.21.15.170:6789/0 conn(0x7f61ac0add40 legacy=0x7f61ac143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:15.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.395+0000 7f61a9d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:16:15.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.395+0000 7f61a9d9b640 1 monclient: found mon.noname-c 2024-03-29T11:16:15.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.395+0000 7f61a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:15.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.395+0000 7f61a9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:16:15.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.395+0000 7f61a9d9b640 1 -- 172.21.15.170:0/3500423698 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac14a6f0 con 0x7f61ac0a9750 2024-03-29T11:16:15.374 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.396+0000 7f61a9d9b640 10 monclient: _finish_auth 0 2024-03-29T11:16:15.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.396+0000 7f61a9d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:16:15.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.396+0000 7f61a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:15.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.396+0000 7f61a9d9b640 1 -- 172.21.15.170:0/3500423698 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f619c001690 con 0x7f61ac0a9750 2024-03-29T11:16:15.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.396+0000 7f61a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:15.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.396+0000 7f61a9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:15.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.396+0000 7f61a9d9b640 10 monclient: dump: 2024-03-29T11:16:15.376 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:15.376 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:15.376 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:15.376 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:15.376 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:15.376 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:15.376 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:15.376 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:15.376 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:15.376 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:15.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.396+0000 7f61a9d9b640 1 -- 172.21.15.170:0/3500423698 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f619c001920 con 0x7f61ac0a9750 2024-03-29T11:16:15.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.396+0000 7f61a9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:15.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.396+0000 7f61a9d9b640 1 -- 172.21.15.170:0/3500423698 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f619c001cb0 con 0x7f61ac0a9750 2024-03-29T11:16:15.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.396+0000 7f61a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:15.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.396+0000 7f61a9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:15.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.396+0000 7f61a9d9b640 10 monclient: dump: 2024-03-29T11:16:15.377 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:15.377 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:15.377 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:15.377 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:15.377 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:15.377 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:15.377 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:15.377 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:15.377 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:15.377 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:15.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.396+0000 7f61b0a29640 5 monclient: authenticate success, global_id 4375 2024-03-29T11:16:15.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.396+0000 7f61b0a29640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:15.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.396+0000 7f61b0a29640 4 set_mon_vals no callback set 2024-03-29T11:16:15.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.397+0000 7f61b0a29640 10 monclient: shutdown 2024-03-29T11:16:15.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.397+0000 7f61b0a29640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:15.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.397+0000 7f61b0a29640 1 -- 172.21.15.170:0/3500423698 >> v1:172.21.15.184:6789/0 conn(0x7f61ac0a9750 legacy=0x7f61ac149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:15.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.397+0000 7f61b0a29640 1 -- 172.21.15.170:0/3500423698 shutdown_connections 2024-03-29T11:16:15.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.397+0000 7f61b0a29640 1 -- 172.21.15.170:0/3500423698 >> 172.21.15.170:0/3500423698 conn(0x7f61ac0673e0 msgr2=0x7f61ac0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:15.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.397+0000 7f61b0a29640 1 -- 172.21.15.170:0/3500423698 shutdown_connections 2024-03-29T11:16:15.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.397+0000 7f61b0a29640 1 -- 172.21.15.170:0/3500423698 wait complete. 2024-03-29T11:16:15.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.398+0000 7f61b0a29640 10 monclient: build_initial_monmap 2024-03-29T11:16:15.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.398+0000 7f61b0a29640 10 monclient: monmap: 2024-03-29T11:16:15.378 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:16:15.378 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:15.378 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:16:15.379 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:16:15.379 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:15.379 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:15.379 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:16:15.379 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:16:15.379 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:16:15.379 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:15.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.399+0000 7f61b0a29640 1 librados: starting msgr at 2024-03-29T11:16:15.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.399+0000 7f61b0a29640 1 librados: starting objecter 2024-03-29T11:16:15.380 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.399+0000 7f61b0a29640 1 Processor -- start 2024-03-29T11:16:15.380 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.399+0000 7f61b0a29640 1 -- start start 2024-03-29T11:16:15.380 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.399+0000 7f61b0a29640 1 librados: setting wanted keys 2024-03-29T11:16:15.380 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.399+0000 7f61b0a29640 1 librados: calling monclient init 2024-03-29T11:16:15.380 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.399+0000 7f61b0a29640 10 monclient: init 2024-03-29T11:16:15.380 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.400+0000 7f61b0a29640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:15.380 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.400+0000 7f61b0a29640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:15.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.406+0000 7f61b0a29640 10 monclient(hunting): picked mon.noname-a con 0x7f61ac0a9750 addr v1:172.21.15.170:6789/0 2024-03-29T11:16:15.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.406+0000 7f61b0a29640 10 monclient(hunting): picked mon.noname-b con 0x7f61ac0aad20 addr v1:172.21.15.184:6789/0 2024-03-29T11:16:15.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.406+0000 7f61b0a29640 10 monclient(hunting): picked mon.noname-c con 0x7f61ac0add40 addr v1:172.21.15.170:6790/0 2024-03-29T11:16:15.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.406+0000 7f61b0a29640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ac146580 con 0x7f61ac0a9750 2024-03-29T11:16:15.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.406+0000 7f61b0a29640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ac06feb0 con 0x7f61ac0aad20 2024-03-29T11:16:15.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.406+0000 7f61b0a29640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ac070190 con 0x7f61ac0add40 2024-03-29T11:16:15.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.406+0000 7f61b0a29640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:15.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.406+0000 7f61b0a29640 10 monclient(hunting): authenticate will time out at 1156.717865s 2024-03-29T11:16:15.389 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.412+0000 7f61aa59c640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f61ac0aad20 0x7f61ac1dbaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:34903/0 (socket says 172.21.15.170:34903) 2024-03-29T11:16:15.389 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.412+0000 7f61aa59c640 1 -- 172.21.15.170:0/266284540 learned_addr learned my addr 172.21.15.170:0/266284540 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:16:15.389 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.412+0000 7f61937fe640 10 client.?.objecter ms_handle_connect 0x7f61ac0aad20 2024-03-29T11:16:15.389 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.412+0000 7f61937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:15.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.415+0000 7f61937fe640 1 -- 172.21.15.170:0/266284540 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252726453 0 0) 0x7f61ac06feb0 con 0x7f61ac0aad20 2024-03-29T11:16:15.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.415+0000 7f61937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:15.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.415+0000 7f61937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:15.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.415+0000 7f61937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:15.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.415+0000 7f61937fe640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:16:15.394 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.415+0000 7f61937fe640 1 -- 172.21.15.170:0/266284540 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6198004eb0 con 0x7f61ac0aad20 2024-03-29T11:16:15.394 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.415+0000 7f61937fe640 1 -- 172.21.15.170:0/266284540 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 584968898 0 0) 0x7f6198004eb0 con 0x7f61ac0aad20 2024-03-29T11:16:15.394 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.415+0000 7f61937fe640 1 -- 172.21.15.170:0/266284540 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6198005300 con 0x7f61ac0aad20 2024-03-29T11:16:15.394 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.415+0000 7f61937fe640 1 -- 172.21.15.170:0/266284540 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f61a00017d0 con 0x7f61ac0aad20 2024-03-29T11:16:15.394 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.415+0000 7f61937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:15.394 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.415+0000 7f61937fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:15.394 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.415+0000 7f61937fe640 10 monclient(hunting): dump: 2024-03-29T11:16:15.394 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:15.394 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:15.394 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:15.394 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:15.394 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:15.394 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:15.395 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:15.395 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:15.395 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:15.395 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:15.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.415+0000 7f61937fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:15.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61937fe640 10 client.?.objecter ms_handle_connect 0x7f61ac0add40 2024-03-29T11:16:15.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:15.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61937fe640 10 client.?.objecter ms_handle_connect 0x7f61ac0a9750 2024-03-29T11:16:15.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:15.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61937fe640 1 -- 172.21.15.170:0/266284540 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2449814099 0 0) 0x7f6198005300 con 0x7f61ac0aad20 2024-03-29T11:16:15.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61937fe640 1 -- 172.21.15.170:0/266284540 >> v1:172.21.15.170:6790/0 conn(0x7f61ac0add40 legacy=0x7f61ac0718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:15.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61937fe640 1 -- 172.21.15.170:0/266284540 >> v1:172.21.15.170:6789/0 conn(0x7f61ac0a9750 legacy=0x7f61ac1db390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:15.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61937fe640 10 monclient: _finish_hunting 0 2024-03-29T11:16:15.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61937fe640 1 monclient: found mon.b 2024-03-29T11:16:15.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:15.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61937fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:16:15.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61937fe640 1 -- 172.21.15.170:0/266284540 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac06e920 con 0x7f61ac0aad20 2024-03-29T11:16:15.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61937fe640 10 monclient: _finish_auth 0 2024-03-29T11:16:15.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61937fe640 10 monclient: _check_auth_tickets 2024-03-29T11:16:15.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:15.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61b0a29640 5 monclient: authenticate success, global_id 4378 2024-03-29T11:16:15.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61b0a29640 10 monclient: _renew_subs 2024-03-29T11:16:15.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61b0a29640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:16:15.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61b0a29640 1 -- 172.21.15.170:0/266284540 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61ac06eb90 con 0x7f61ac0aad20 2024-03-29T11:16:15.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61b0a29640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:15.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61b0a29640 10 monclient: _renew_subs 2024-03-29T11:16:15.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61b0a29640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:16:15.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61b0a29640 1 -- 172.21.15.170:0/266284540 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61ac06df10 con 0x7f61ac0aad20 2024-03-29T11:16:15.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.416+0000 7f61b0a29640 1 librados: init done 2024-03-29T11:16:15.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.417+0000 7f61937fe640 1 -- 172.21.15.170:0/266284540 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61a0004800 con 0x7f61ac0aad20 2024-03-29T11:16:15.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.417+0000 7f61937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:15.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.417+0000 7f61937fe640 1 -- 172.21.15.170:0/266284540 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f61a00017d0 con 0x7f61ac0aad20 2024-03-29T11:16:15.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.417+0000 7f61937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:15.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.417+0000 7f61937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:15.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.417+0000 7f61937fe640 10 monclient: dump: 2024-03-29T11:16:15.398 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:15.398 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:15.398 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:15.399 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:15.399 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:15.399 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:15.399 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:15.399 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:15.399 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:15.399 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:15.399 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.418+0000 7f61a8d99640 4 set_mon_vals no callback set 2024-03-29T11:16:15.399 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.419+0000 7f61937fe640 1 -- 172.21.15.170:0/266284540 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f61a001a7e0 con 0x7f61ac0aad20 2024-03-29T11:16:15.399 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.419+0000 7f61b0a29640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:15.399 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.419+0000 7f61b0a29640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:15.399 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.419+0000 7f61b0a29640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:16:15.399 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.419+0000 7f61b0a29640 1 -- 172.21.15.170:0/266284540 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f618c005080 con 0x7f61ac0aad20 2024-03-29T11:16:15.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.419+0000 7f61937fe640 1 -- 172.21.15.170:0/266284540 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f61a0054350 con 0x7f61ac0aad20 2024-03-29T11:16:15.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.419+0000 7f61937fe640 10 client.4378.objecter ms_dispatch 0x7f61ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:15.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.419+0000 7f61937fe640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:15.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.419+0000 7f61937fe640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:15.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.419+0000 7f61937fe640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-29T11:16:15.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.420+0000 7f61aad9d640 10 monclient: get_auth_request con 0x7f619806e220 auth_method 0 2024-03-29T11:16:15.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.420+0000 7f61937fe640 10 client.4378.objecter ms_handle_connect 0x7f619806e220 2024-03-29T11:16:15.403 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.427+0000 7f61937fe640 1 -- 172.21.15.170:0/266284540 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f61a0053a20 con 0x7f61ac0aad20 2024-03-29T11:16:15.403 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.427+0000 7f61937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:15.403 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.427+0000 7f61937fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:15.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.788+0000 7f61b0a29640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:15.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.788+0000 7f61b0a29640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:15.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.788+0000 7f61b0a29640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:16:15.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.788+0000 7f61b0a29640 1 -- 172.21.15.170:0/266284540 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f618c005330 con 0x7f61ac0aad20 2024-03-29T11:16:15.766 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:16:15.767 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:15.767 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.789+0000 7f61937fe640 1 -- 172.21.15.170:0/266284540 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f61a0053a20 con 0x7f61ac0aad20 2024-03-29T11:16:15.767 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.789+0000 7f61937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:15.767 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.789+0000 7f61937fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:15.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.793+0000 7f61b0a29640 10 librados: watch_flush enter 2024-03-29T11:16:15.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.793+0000 7f61b0a29640 10 librados: watch_flush exit 2024-03-29T11:16:15.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.793+0000 7f61b0a29640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-29T11:16:15.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.793+0000 7f61b0a29640 10 client.4378.objecter successfully canceled tick 2024-03-29T11:16:15.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.793+0000 7f61b0a29640 1 -- 172.21.15.170:0/266284540 >> v1:172.21.15.170:6816/3298994444 conn(0x7f619806e220 legacy=0x7f61980706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:15.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.793+0000 7f61b0a29640 10 monclient: shutdown 2024-03-29T11:16:15.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.793+0000 7f61b0a29640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:15.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.793+0000 7f61b0a29640 1 -- 172.21.15.170:0/266284540 >> v1:172.21.15.184:6789/0 conn(0x7f61ac0aad20 legacy=0x7f61ac1dbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:15.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.793+0000 7f61b0a29640 1 -- 172.21.15.170:0/266284540 shutdown_connections 2024-03-29T11:16:15.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.793+0000 7f61b0a29640 1 -- 172.21.15.170:0/266284540 >> 172.21.15.170:0/266284540 conn(0x7f61ac0673e0 msgr2=0x7f61ac0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:15.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.793+0000 7f61b0a29640 1 -- 172.21.15.170:0/266284540 shutdown_connections 2024-03-29T11:16:15.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.793+0000 7f61b0a29640 1 -- 172.21.15.170:0/266284540 wait complete. 2024-03-29T11:16:15.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:15.794+0000 7f61b0a29640 1 librados: shutdown 2024-03-29T11:16:15.795 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:22.602 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:22.758 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.781+0000 7f949e867640 10 monclient: get_monmap_and_config 2024-03-29T11:16:22.759 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.781+0000 7f949e867640 10 monclient: build_initial_monmap 2024-03-29T11:16:22.759 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.781+0000 7f949e867640 10 monclient: monmap: 2024-03-29T11:16:22.759 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:16:22.759 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:22.759 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:16:22.782348+0000 2024-03-29T11:16:22.759 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:16:22.782348+0000 2024-03-29T11:16:22.759 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:22.759 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:22.759 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:16:22.759 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:16:22.759 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:16:22.759 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:22.761 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.784+0000 7f949e867640 1 Processor -- start 2024-03-29T11:16:22.761 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.784+0000 7f949e867640 1 -- start start 2024-03-29T11:16:22.761 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.784+0000 7f949e867640 10 monclient: init 2024-03-29T11:16:22.764 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.785+0000 7f949e867640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:22.764 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.785+0000 7f949e867640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:16:22.764 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.785+0000 7f949e867640 10 monclient(hunting): picked mon.noname-b con 0x7f9490006d00 addr v1:172.21.15.170:6790/0 2024-03-29T11:16:22.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.785+0000 7f949e867640 10 monclient(hunting): picked mon.noname-a con 0x7f9490031c70 addr v1:172.21.15.170:6789/0 2024-03-29T11:16:22.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.785+0000 7f949e867640 10 monclient(hunting): picked mon.noname-c con 0x7f94900c4fb0 addr v1:172.21.15.184:6789/0 2024-03-29T11:16:22.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.785+0000 7f949e867640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9490032340 con 0x7f9490031c70 2024-03-29T11:16:22.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.785+0000 7f949e867640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9490033cc0 con 0x7f94900c4fb0 2024-03-29T11:16:22.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.785+0000 7f949e867640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9490032f00 con 0x7f9490006d00 2024-03-29T11:16:22.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.785+0000 7f949e867640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:22.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.785+0000 7f949e867640 10 monclient(hunting): authenticate will time out at 1164.096794s 2024-03-29T11:16:22.767 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.791+0000 7f949e066640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f94900c4fb0 0x7f94900d3100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:57065/0 (socket says 172.21.15.170:57065) 2024-03-29T11:16:22.767 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.791+0000 7f949e066640 1 -- 172.21.15.170:0/4005659372 learned_addr learned my addr 172.21.15.170:0/4005659372 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:16:22.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.793+0000 7f949c863640 1 -- 172.21.15.170:0/4005659372 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608749078 0 0) 0x7f9490032340 con 0x7f9490031c70 2024-03-29T11:16:22.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.793+0000 7f949c863640 10 monclient(hunting): _negotiate 2024-03-29T11:16:22.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.793+0000 7f949c863640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:22.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.793+0000 7f949c863640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:22.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.793+0000 7f949c863640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:16:22.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.793+0000 7f949c863640 1 -- 172.21.15.170:0/4005659372 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9484003520 con 0x7f9490031c70 2024-03-29T11:16:22.773 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.794+0000 7f949c863640 1 -- 172.21.15.170:0/4005659372 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1440652055 0 0) 0x7f9484003520 con 0x7f9490031c70 2024-03-29T11:16:22.774 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.794+0000 7f949c863640 1 -- 172.21.15.170:0/4005659372 >> v1:172.21.15.170:6790/0 conn(0x7f9490006d00 legacy=0x7f94900327f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:22.774 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.794+0000 7f949c863640 1 -- 172.21.15.170:0/4005659372 >> v1:172.21.15.184:6789/0 conn(0x7f94900c4fb0 legacy=0x7f94900d3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:22.774 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.794+0000 7f949c863640 10 monclient: _finish_hunting 0 2024-03-29T11:16:22.774 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.794+0000 7f949c863640 1 monclient: found mon.noname-a 2024-03-29T11:16:22.774 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.794+0000 7f949c863640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:22.774 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.794+0000 7f949c863640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:16:22.774 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.794+0000 7f949c863640 1 -- 172.21.15.170:0/4005659372 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94900307f0 con 0x7f9490031c70 2024-03-29T11:16:22.774 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.794+0000 7f949c863640 10 monclient: _finish_auth 0 2024-03-29T11:16:22.774 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.794+0000 7f949c863640 10 monclient: _check_auth_tickets 2024-03-29T11:16:22.774 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.794+0000 7f949c863640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:22.774 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.794+0000 7f949c863640 1 -- 172.21.15.170:0/4005659372 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9494001690 con 0x7f9490031c70 2024-03-29T11:16:22.774 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.794+0000 7f949c863640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:22.775 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.794+0000 7f949c863640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:22.775 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.794+0000 7f949c863640 10 monclient: dump: 2024-03-29T11:16:22.775 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:22.775 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:22.775 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:22.775 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:22.775 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:22.775 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:22.775 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:22.775 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:22.775 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:22.775 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:22.775 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.795+0000 7f949e867640 5 monclient: authenticate success, global_id 4389 2024-03-29T11:16:22.776 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.795+0000 7f949e867640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:22.776 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.795+0000 7f949c863640 1 -- 172.21.15.170:0/4005659372 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9494001860 con 0x7f9490031c70 2024-03-29T11:16:22.776 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.795+0000 7f949c863640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:22.776 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.795+0000 7f949e867640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:22.776 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.795+0000 7f949e867640 4 set_mon_vals no callback set 2024-03-29T11:16:22.776 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.796+0000 7f949c863640 1 -- 172.21.15.170:0/4005659372 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9494001cb0 con 0x7f9490031c70 2024-03-29T11:16:22.776 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.796+0000 7f949c863640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:22.776 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.796+0000 7f949c863640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:22.776 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.796+0000 7f949c863640 10 monclient: dump: 2024-03-29T11:16:22.776 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:22.776 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:22.776 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:22.776 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:22.776 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:22.777 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:22.777 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:22.777 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:22.777 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:22.777 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:22.777 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.796+0000 7f949e867640 10 monclient: shutdown 2024-03-29T11:16:22.777 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.796+0000 7f949e867640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:22.777 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.796+0000 7f949e867640 1 -- 172.21.15.170:0/4005659372 >> v1:172.21.15.170:6789/0 conn(0x7f9490031c70 legacy=0x7f94900c48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:22.777 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.797+0000 7f949e867640 1 -- 172.21.15.170:0/4005659372 shutdown_connections 2024-03-29T11:16:22.777 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.797+0000 7f949e867640 1 -- 172.21.15.170:0/4005659372 >> 172.21.15.170:0/4005659372 conn(0x7f9490005c40 msgr2=0x7f9490006050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:22.779 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.798+0000 7f949e867640 1 -- 172.21.15.170:0/4005659372 shutdown_connections 2024-03-29T11:16:22.779 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.798+0000 7f949e867640 1 -- 172.21.15.170:0/4005659372 wait complete. 2024-03-29T11:16:22.779 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.799+0000 7f949e867640 10 monclient: build_initial_monmap 2024-03-29T11:16:22.780 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.799+0000 7f949e867640 10 monclient: monmap: 2024-03-29T11:16:22.780 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:16:22.780 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:22.780 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:16:22.780 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:16:22.780 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:22.780 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:22.780 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:16:22.780 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:16:22.780 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:16:22.780 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:22.780 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.799+0000 7f949e867640 1 librados: starting msgr at 2024-03-29T11:16:22.780 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.799+0000 7f949e867640 1 librados: starting objecter 2024-03-29T11:16:22.781 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.799+0000 7f949e867640 1 Processor -- start 2024-03-29T11:16:22.781 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.799+0000 7f949e867640 1 -- start start 2024-03-29T11:16:22.781 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.799+0000 7f949e867640 1 librados: setting wanted keys 2024-03-29T11:16:22.781 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.799+0000 7f949e867640 1 librados: calling monclient init 2024-03-29T11:16:22.781 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.799+0000 7f949e867640 10 monclient: init 2024-03-29T11:16:22.781 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.800+0000 7f949e867640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:22.781 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.800+0000 7f949e867640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:16:22.781 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.800+0000 7f949e867640 10 monclient(hunting): picked mon.noname-c con 0x7f9490006d00 addr v1:172.21.15.170:6790/0 2024-03-29T11:16:22.781 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.800+0000 7f949e867640 10 monclient(hunting): picked mon.noname-b con 0x7f9490031c70 addr v1:172.21.15.184:6789/0 2024-03-29T11:16:22.781 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.800+0000 7f949e867640 10 monclient(hunting): picked mon.noname-a con 0x7f94900c4fb0 addr v1:172.21.15.170:6789/0 2024-03-29T11:16:22.781 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.800+0000 7f949e867640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94900c3a40 con 0x7f94900c4fb0 2024-03-29T11:16:22.781 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.800+0000 7f949e867640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94900c9090 con 0x7f9490031c70 2024-03-29T11:16:22.781 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.800+0000 7f949e867640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94900c2a40 con 0x7f9490006d00 2024-03-29T11:16:22.781 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.800+0000 7f949e867640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:22.782 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.800+0000 7f949e867640 10 monclient(hunting): authenticate will time out at 1164.111498s 2024-03-29T11:16:22.782 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.804+0000 7f949e066640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f94900c4fb0 0x7f949016bdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:40411/0 (socket says 172.21.15.170:40411) 2024-03-29T11:16:22.782 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.804+0000 7f949e066640 1 -- 172.21.15.170:0/775656432 learned_addr learned my addr 172.21.15.170:0/775656432 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:16:22.784 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.805+0000 7f948e7fc640 10 client.?.objecter ms_handle_connect 0x7f94900c4fb0 2024-03-29T11:16:22.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:22.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 10 client.?.objecter ms_handle_connect 0x7f9490006d00 2024-03-29T11:16:22.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:22.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 10 client.?.objecter ms_handle_connect 0x7f9490031c70 2024-03-29T11:16:22.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:22.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405319942 0 0) 0x7f94900c3a40 con 0x7f94900c4fb0 2024-03-29T11:16:22.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:22.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:22.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:22.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:16:22.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f947c004eb0 con 0x7f94900c4fb0 2024-03-29T11:16:22.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914211588 0 0) 0x7f94900c9090 con 0x7f9490031c70 2024-03-29T11:16:22.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:22.786 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:22.786 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:22.786 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:16:22.786 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94900c3a40 con 0x7f9490031c70 2024-03-29T11:16:22.786 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 490372790 0 0) 0x7f947c004eb0 con 0x7f94900c4fb0 2024-03-29T11:16:22.786 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f947c0064f0 con 0x7f94900c4fb0 2024-03-29T11:16:22.786 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9488001870 con 0x7f94900c4fb0 2024-03-29T11:16:22.786 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:22.786 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:22.786 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 10 monclient(hunting): dump: 2024-03-29T11:16:22.786 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:22.786 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:22.786 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:22.786 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:22.787 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:22.787 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:22.787 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:22.787 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:22.787 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:22.787 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:22.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.806+0000 7f948e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:22.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1469437476 0 0) 0x7f94900c3a40 con 0x7f9490031c70 2024-03-29T11:16:22.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f947c004eb0 con 0x7f9490031c70 2024-03-29T11:16:22.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9494001b70 con 0x7f9490031c70 2024-03-29T11:16:22.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f948e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:22.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f948e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:22.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f948e7fc640 10 monclient(hunting): dump: 2024-03-29T11:16:22.787 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:22.788 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:22.788 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:22.788 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:22.788 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:22.788 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:22.788 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:22.788 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:22.788 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:22.788 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:22.788 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 852338052 0 0) 0x7f947c0064f0 con 0x7f94900c4fb0 2024-03-29T11:16:22.788 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 >> v1:172.21.15.170:6790/0 conn(0x7f9490006d00 legacy=0x7f94900c2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:22.788 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 >> v1:172.21.15.184:6789/0 conn(0x7f9490031c70 legacy=0x7f94900381d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:22.788 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f948e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:16:22.788 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f948e7fc640 1 monclient: found mon.a 2024-03-29T11:16:22.789 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f948e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:22.789 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f948e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:22.789 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94900388e0 con 0x7f94900c4fb0 2024-03-29T11:16:22.789 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f948e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:16:22.789 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f948e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:16:22.789 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f948e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:22.791 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f949e867640 5 monclient: authenticate success, global_id 4392 2024-03-29T11:16:22.791 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f949e867640 10 monclient: _renew_subs 2024-03-29T11:16:22.791 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f949e867640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:22.791 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f949e867640 1 -- 172.21.15.170:0/775656432 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94900ca7a0 con 0x7f94900c4fb0 2024-03-29T11:16:22.791 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f949e867640 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:22.791 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f949e867640 10 monclient: _renew_subs 2024-03-29T11:16:22.791 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f949e867640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:22.791 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f949e867640 1 -- 172.21.15.170:0/775656432 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94900aec50 con 0x7f94900c4fb0 2024-03-29T11:16:22.791 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.807+0000 7f949e867640 1 librados: init done 2024-03-29T11:16:22.791 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.809+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9488001cc0 con 0x7f94900c4fb0 2024-03-29T11:16:22.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.809+0000 7f948e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:22.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.809+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f94880049c0 con 0x7f94900c4fb0 2024-03-29T11:16:22.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.809+0000 7f948e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:22.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.809+0000 7f948e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:22.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.809+0000 7f948e7fc640 10 monclient: dump: 2024-03-29T11:16:22.792 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:22.792 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:22.792 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:22.792 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:22.792 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:22.792 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:22.792 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:22.792 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:22.792 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:22.792 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:22.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.809+0000 7f948ffff640 4 set_mon_vals no callback set 2024-03-29T11:16:22.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.809+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f9488004be0 con 0x7f94900c4fb0 2024-03-29T11:16:22.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.810+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f9488084980 con 0x7f94900c4fb0 2024-03-29T11:16:22.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.810+0000 7f948e7fc640 10 client.4392.objecter ms_dispatch 0x7f9498000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:22.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.810+0000 7f948e7fc640 3 client.4392.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:22.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.810+0000 7f948e7fc640 3 client.4392.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:22.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.811+0000 7f948e7fc640 20 client.4392.objecter dump_active .. 0 homeless 2024-03-29T11:16:22.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.812+0000 7f949d865640 10 monclient: get_auth_request con 0x7f947c06e250 auth_method 0 2024-03-29T11:16:22.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.812+0000 7f948e7fc640 10 client.4392.objecter ms_handle_connect 0x7f947c06e250 2024-03-29T11:16:22.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.813+0000 7f9483fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:22.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.813+0000 7f9483fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:22.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.813+0000 7f9483fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:22.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.813+0000 7f9483fff640 1 -- 172.21.15.170:0/775656432 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9478005080 con 0x7f94900c4fb0 2024-03-29T11:16:22.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.823+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9488050260 con 0x7f94900c4fb0 2024-03-29T11:16:22.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.823+0000 7f948e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:22.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:22.823+0000 7f948e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:23.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.147+0000 7f9483fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:23.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.147+0000 7f9483fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:23.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.147+0000 7f9483fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:23.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.147+0000 7f9483fff640 1 -- 172.21.15.170:0/775656432 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9478005330 con 0x7f94900c4fb0 2024-03-29T11:16:23.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.148+0000 7f948e7fc640 1 -- 172.21.15.170:0/775656432 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f9488052190 con 0x7f94900c4fb0 2024-03-29T11:16:23.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.148+0000 7f948e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:23.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.148+0000 7f948e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:23.125 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:16:23.125 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:23.128 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.151+0000 7f9483fff640 10 librados: watch_flush enter 2024-03-29T11:16:23.129 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.152+0000 7f9483fff640 10 librados: watch_flush exit 2024-03-29T11:16:23.129 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.152+0000 7f9483fff640 20 client.4392.objecter shutdown clearing up homeless session... 2024-03-29T11:16:23.129 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.152+0000 7f9483fff640 10 client.4392.objecter successfully canceled tick 2024-03-29T11:16:23.129 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.152+0000 7f9483fff640 1 -- 172.21.15.170:0/775656432 >> v1:172.21.15.170:6816/3298994444 conn(0x7f947c06e250 legacy=0x7f947c070710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:23.129 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.152+0000 7f9483fff640 10 monclient: shutdown 2024-03-29T11:16:23.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.152+0000 7f9483fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:23.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.152+0000 7f9483fff640 1 -- 172.21.15.170:0/775656432 >> v1:172.21.15.170:6789/0 conn(0x7f94900c4fb0 legacy=0x7f949016bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:23.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.154+0000 7f9483fff640 1 -- 172.21.15.170:0/775656432 shutdown_connections 2024-03-29T11:16:23.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.154+0000 7f9483fff640 1 -- 172.21.15.170:0/775656432 >> 172.21.15.170:0/775656432 conn(0x7f9490005c40 msgr2=0x7f94900043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:23.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.154+0000 7f9483fff640 1 -- 172.21.15.170:0/775656432 shutdown_connections 2024-03-29T11:16:23.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.154+0000 7f9483fff640 1 -- 172.21.15.170:0/775656432 wait complete. 2024-03-29T11:16:23.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:23.154+0000 7f9483fff640 1 librados: shutdown 2024-03-29T11:16:23.148 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:29.954 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:30.099 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.122+0000 7fc93fcba640 10 monclient: get_monmap_and_config 2024-03-29T11:16:30.100 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.122+0000 7fc93fcba640 10 monclient: build_initial_monmap 2024-03-29T11:16:30.100 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.122+0000 7fc93fcba640 10 monclient: monmap: 2024-03-29T11:16:30.100 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:16:30.100 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:30.100 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:16:30.123603+0000 2024-03-29T11:16:30.100 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:16:30.123603+0000 2024-03-29T11:16:30.100 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:30.100 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:30.100 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:16:30.100 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:16:30.100 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:16:30.100 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:30.101 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.123+0000 7fc93fcba640 1 Processor -- start 2024-03-29T11:16:30.101 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.123+0000 7fc93fcba640 1 -- start start 2024-03-29T11:16:30.101 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.123+0000 7fc93fcba640 10 monclient: init 2024-03-29T11:16:30.102 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.124+0000 7fc93fcba640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:30.102 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.124+0000 7fc93fcba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:30.102 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.124+0000 7fc93fcba640 10 monclient(hunting): picked mon.noname-c con 0x7fc9380b0770 addr v1:172.21.15.184:6789/0 2024-03-29T11:16:30.102 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.124+0000 7fc93fcba640 10 monclient(hunting): picked mon.noname-b con 0x7fc93814a700 addr v1:172.21.15.170:6790/0 2024-03-29T11:16:30.102 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.124+0000 7fc93fcba640 10 monclient(hunting): picked mon.noname-a con 0x7fc9380ac5a0 addr v1:172.21.15.170:6789/0 2024-03-29T11:16:30.102 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.124+0000 7fc93fcba640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9380aba70 con 0x7fc9380ac5a0 2024-03-29T11:16:30.103 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.124+0000 7fc93fcba640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9380aecb0 con 0x7fc9380b0770 2024-03-29T11:16:30.103 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.124+0000 7fc93fcba640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9380af150 con 0x7fc93814a700 2024-03-29T11:16:30.103 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.124+0000 7fc93fcba640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:30.103 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.124+0000 7fc93fcba640 10 monclient(hunting): authenticate will time out at 1171.436646s 2024-03-29T11:16:30.103 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.125+0000 7fc93d255640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fc93814a700 0x7fc9380abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:39304/0 (socket says 172.21.15.170:39304) 2024-03-29T11:16:30.103 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.125+0000 7fc93d255640 1 -- 172.21.15.170:0/2612716818 learned_addr learned my addr 172.21.15.170:0/2612716818 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:16:30.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.125+0000 7fc93ca54640 1 -- 172.21.15.170:0/2612716818 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251114196 0 0) 0x7fc9380af150 con 0x7fc93814a700 2024-03-29T11:16:30.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.125+0000 7fc93ca54640 10 monclient(hunting): _negotiate 2024-03-29T11:16:30.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.125+0000 7fc93ca54640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:30.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.125+0000 7fc93ca54640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:30.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.125+0000 7fc93ca54640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:16:30.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.125+0000 7fc93ca54640 1 -- 172.21.15.170:0/2612716818 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc920003520 con 0x7fc93814a700 2024-03-29T11:16:30.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.125+0000 7fc93ca54640 1 -- 172.21.15.170:0/2612716818 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692607323 0 0) 0x7fc9380aba70 con 0x7fc9380ac5a0 2024-03-29T11:16:30.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.125+0000 7fc93ca54640 10 monclient(hunting): _negotiate 2024-03-29T11:16:30.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.125+0000 7fc93ca54640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:30.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.125+0000 7fc93ca54640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:30.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.125+0000 7fc93ca54640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:16:30.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.125+0000 7fc93ca54640 1 -- 172.21.15.170:0/2612716818 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9380af150 con 0x7fc9380ac5a0 2024-03-29T11:16:30.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 1 -- 172.21.15.170:0/2612716818 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 867984937 0 0) 0x7fc9380aecb0 con 0x7fc9380b0770 2024-03-29T11:16:30.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 10 monclient(hunting): _negotiate 2024-03-29T11:16:30.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:30.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:30.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:16:30.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 1 -- 172.21.15.170:0/2612716818 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9380aba70 con 0x7fc9380b0770 2024-03-29T11:16:30.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 1 -- 172.21.15.170:0/2612716818 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4221014275 0 0) 0x7fc9380af150 con 0x7fc9380ac5a0 2024-03-29T11:16:30.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 1 -- 172.21.15.170:0/2612716818 >> v1:172.21.15.170:6790/0 conn(0x7fc93814a700 legacy=0x7fc9380abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:30.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 1 -- 172.21.15.170:0/2612716818 >> v1:172.21.15.184:6789/0 conn(0x7fc9380b0770 legacy=0x7fc938149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:30.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 10 monclient: _finish_hunting 0 2024-03-29T11:16:30.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 1 monclient: found mon.noname-a 2024-03-29T11:16:30.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:30.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:16:30.108 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 1 -- 172.21.15.170:0/2612716818 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9380ac9d0 con 0x7fc9380ac5a0 2024-03-29T11:16:30.108 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 10 monclient: _finish_auth 0 2024-03-29T11:16:30.108 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 10 monclient: _check_auth_tickets 2024-03-29T11:16:30.108 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:30.108 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 1 -- 172.21.15.170:0/2612716818 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fc92c001670 con 0x7fc9380ac5a0 2024-03-29T11:16:30.108 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:30.108 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:30.108 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93ca54640 10 monclient: dump: 2024-03-29T11:16:30.108 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:30.108 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:30.108 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:30.108 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:30.108 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:30.108 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:30.108 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:30.109 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:30.109 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:30.109 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:30.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93fcba640 5 monclient: authenticate success, global_id 4395 2024-03-29T11:16:30.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.126+0000 7fc93fcba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:30.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.127+0000 7fc93ca54640 1 -- 172.21.15.170:0/2612716818 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc92c0017d0 con 0x7fc9380ac5a0 2024-03-29T11:16:30.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.127+0000 7fc93ca54640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:30.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.127+0000 7fc93fcba640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:30.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.127+0000 7fc93fcba640 4 set_mon_vals no callback set 2024-03-29T11:16:30.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.127+0000 7fc93fcba640 10 monclient: shutdown 2024-03-29T11:16:30.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.127+0000 7fc93fcba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:30.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.127+0000 7fc93fcba640 1 -- 172.21.15.170:0/2612716818 >> v1:172.21.15.170:6789/0 conn(0x7fc9380ac5a0 legacy=0x7fc938151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:30.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.128+0000 7fc93fcba640 1 -- 172.21.15.170:0/2612716818 shutdown_connections 2024-03-29T11:16:30.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.128+0000 7fc93fcba640 1 -- 172.21.15.170:0/2612716818 >> 172.21.15.170:0/2612716818 conn(0x7fc93806b660 msgr2=0x7fc93806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:30.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.128+0000 7fc93fcba640 1 -- 172.21.15.170:0/2612716818 shutdown_connections 2024-03-29T11:16:30.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.128+0000 7fc93fcba640 1 -- 172.21.15.170:0/2612716818 wait complete. 2024-03-29T11:16:30.112 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.129+0000 7fc93fcba640 10 monclient: build_initial_monmap 2024-03-29T11:16:30.113 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.129+0000 7fc93fcba640 10 monclient: monmap: 2024-03-29T11:16:30.113 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:16:30.113 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:30.113 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:16:30.113 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:16:30.113 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:30.113 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:30.113 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:16:30.113 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:16:30.113 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:16:30.113 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:30.113 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.129+0000 7fc93fcba640 1 librados: starting msgr at 2024-03-29T11:16:30.113 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.129+0000 7fc93fcba640 1 librados: starting objecter 2024-03-29T11:16:30.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.130+0000 7fc93fcba640 1 Processor -- start 2024-03-29T11:16:30.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.130+0000 7fc93fcba640 1 -- start start 2024-03-29T11:16:30.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.130+0000 7fc93fcba640 1 librados: setting wanted keys 2024-03-29T11:16:30.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.130+0000 7fc93fcba640 1 librados: calling monclient init 2024-03-29T11:16:30.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.130+0000 7fc93fcba640 10 monclient: init 2024-03-29T11:16:30.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.130+0000 7fc93fcba640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:30.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.130+0000 7fc93fcba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:30.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.130+0000 7fc93fcba640 10 monclient(hunting): picked mon.noname-b con 0x7fc9380ac5a0 addr v1:172.21.15.184:6789/0 2024-03-29T11:16:30.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.130+0000 7fc93fcba640 10 monclient(hunting): picked mon.noname-c con 0x7fc9380b0770 addr v1:172.21.15.170:6790/0 2024-03-29T11:16:30.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.130+0000 7fc93fcba640 10 monclient(hunting): picked mon.noname-a con 0x7fc93814a700 addr v1:172.21.15.170:6789/0 2024-03-29T11:16:30.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.130+0000 7fc93fcba640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9381239d0 con 0x7fc93814a700 2024-03-29T11:16:30.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.130+0000 7fc93fcba640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc938126510 con 0x7fc9380ac5a0 2024-03-29T11:16:30.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.130+0000 7fc93fcba640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9381285c0 con 0x7fc9380b0770 2024-03-29T11:16:30.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.130+0000 7fc93fcba640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:30.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.130+0000 7fc93fcba640 10 monclient(hunting): authenticate will time out at 1171.443001s 2024-03-29T11:16:30.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc93da56640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fc9380ac5a0 0x7fc938144ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:51338/0 (socket says 172.21.15.170:51338) 2024-03-29T11:16:30.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc93da56640 1 -- 172.21.15.170:0/3201463208 learned_addr learned my addr 172.21.15.170:0/3201463208 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:16:30.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 client.?.objecter ms_handle_connect 0x7fc9380ac5a0 2024-03-29T11:16:30.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:30.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 client.?.objecter ms_handle_connect 0x7fc93814a700 2024-03-29T11:16:30.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:30.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131179452 0 0) 0x7fc938126510 con 0x7fc9380ac5a0 2024-03-29T11:16:30.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:30.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:30.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:30.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:16:30.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc924004eb0 con 0x7fc9380ac5a0 2024-03-29T11:16:30.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 client.?.objecter ms_handle_connect 0x7fc9380b0770 2024-03-29T11:16:30.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:30.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740938072 0 0) 0x7fc9381239d0 con 0x7fc93814a700 2024-03-29T11:16:30.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:30.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:30.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:30.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:16:30.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc938126510 con 0x7fc93814a700 2024-03-29T11:16:30.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180248897 0 0) 0x7fc9381285c0 con 0x7fc9380b0770 2024-03-29T11:16:30.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:30.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:30.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:30.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:16:30.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9381239d0 con 0x7fc9380b0770 2024-03-29T11:16:30.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1293196142 0 0) 0x7fc924004eb0 con 0x7fc9380ac5a0 2024-03-29T11:16:30.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9381285c0 con 0x7fc9380ac5a0 2024-03-29T11:16:30.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fc930001a30 con 0x7fc9380ac5a0 2024-03-29T11:16:30.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:30.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:30.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 monclient(hunting): dump: 2024-03-29T11:16:30.117 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:30.117 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:30.117 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:30.117 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:30.117 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:30.117 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:30.117 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:30.117 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:30.117 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:30.117 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:30.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.131+0000 7fc9367fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:30.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4012358922 0 0) 0x7fc938126510 con 0x7fc93814a700 2024-03-29T11:16:30.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc924004eb0 con 0x7fc93814a700 2024-03-29T11:16:30.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fc92c003790 con 0x7fc93814a700 2024-03-29T11:16:30.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc9367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:30.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc9367fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:30.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc9367fc640 10 monclient(hunting): dump: 2024-03-29T11:16:30.118 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:30.118 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:30.118 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:30.118 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:30.118 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:30.118 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:30.118 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:30.118 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:30.118 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:30.118 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:30.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2444911587 0 0) 0x7fc9381285c0 con 0x7fc9380ac5a0 2024-03-29T11:16:30.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 >> v1:172.21.15.170:6790/0 conn(0x7fc9380b0770 legacy=0x7fc9381269a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:30.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 >> v1:172.21.15.170:6789/0 conn(0x7fc93814a700 legacy=0x7fc9381e7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:30.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc9367fc640 10 monclient: _finish_hunting 0 2024-03-29T11:16:30.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc9367fc640 1 monclient: found mon.b 2024-03-29T11:16:30.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc9367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:30.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc9367fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:16:30.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9381dfa30 con 0x7fc9380ac5a0 2024-03-29T11:16:30.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc9367fc640 10 monclient: _finish_auth 0 2024-03-29T11:16:30.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc9367fc640 10 monclient: _check_auth_tickets 2024-03-29T11:16:30.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc9367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:30.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc93fcba640 5 monclient: authenticate success, global_id 4390 2024-03-29T11:16:30.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc93fcba640 10 monclient: _renew_subs 2024-03-29T11:16:30.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc93fcba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:16:30.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc93fcba640 1 -- 172.21.15.170:0/3201463208 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9381dfc80 con 0x7fc9380ac5a0 2024-03-29T11:16:30.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc93fcba640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:30.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc93fcba640 10 monclient: _renew_subs 2024-03-29T11:16:30.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc93fcba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:16:30.120 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc93fcba640 1 -- 172.21.15.170:0/3201463208 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc938126190 con 0x7fc9380ac5a0 2024-03-29T11:16:30.120 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.132+0000 7fc93fcba640 1 librados: init done 2024-03-29T11:16:30.120 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.133+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc930001c70 con 0x7fc9380ac5a0 2024-03-29T11:16:30.120 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.133+0000 7fc9367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:30.120 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.133+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fc930003a60 con 0x7fc9380ac5a0 2024-03-29T11:16:30.120 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.133+0000 7fc9367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:30.120 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.133+0000 7fc9367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:30.120 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.133+0000 7fc9367fc640 10 monclient: dump: 2024-03-29T11:16:30.120 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:30.120 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:30.120 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:30.120 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:30.120 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:30.120 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:30.120 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:30.120 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:30.120 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:30.121 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:30.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.133+0000 7fc937fff640 4 set_mon_vals no callback set 2024-03-29T11:16:30.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.134+0000 7fc93fcba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:30.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.134+0000 7fc93fcba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:30.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.134+0000 7fc93fcba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:16:30.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.134+0000 7fc93fcba640 1 -- 172.21.15.170:0/3201463208 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc918005080 con 0x7fc9380ac5a0 2024-03-29T11:16:30.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.135+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fc930003bc0 con 0x7fc9380ac5a0 2024-03-29T11:16:30.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.136+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fc930084ad0 con 0x7fc9380ac5a0 2024-03-29T11:16:30.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.136+0000 7fc9367fc640 10 client.4390.objecter ms_dispatch 0x7fc938000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:30.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.136+0000 7fc9367fc640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:30.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.136+0000 7fc9367fc640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:30.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.136+0000 7fc9367fc640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-29T11:16:30.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.136+0000 7fc93d255640 10 monclient: get_auth_request con 0x7fc92406dfe0 auth_method 0 2024-03-29T11:16:30.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.137+0000 7fc9367fc640 10 client.4390.objecter ms_handle_connect 0x7fc92406dfe0 2024-03-29T11:16:30.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.144+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9300503b0 con 0x7fc9380ac5a0 2024-03-29T11:16:30.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.144+0000 7fc9367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:30.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.144+0000 7fc9367fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:30.499 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.521+0000 7fc93fcba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:30.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.522+0000 7fc93fcba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:30.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.522+0000 7fc93fcba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:16:30.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.522+0000 7fc93fcba640 1 -- 172.21.15.170:0/3201463208 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc918005330 con 0x7fc9380ac5a0 2024-03-29T11:16:30.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.523+0000 7fc9367fc640 1 -- 172.21.15.170:0/3201463208 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fc9300522e0 con 0x7fc9380ac5a0 2024-03-29T11:16:30.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.523+0000 7fc9367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:30.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.523+0000 7fc9367fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:30.501 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:16:30.502 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:30.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.526+0000 7fc93fcba640 10 librados: watch_flush enter 2024-03-29T11:16:30.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.527+0000 7fc93fcba640 10 librados: watch_flush exit 2024-03-29T11:16:30.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.527+0000 7fc93fcba640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-29T11:16:30.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.527+0000 7fc93fcba640 10 client.4390.objecter successfully canceled tick 2024-03-29T11:16:30.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.527+0000 7fc93fcba640 1 -- 172.21.15.170:0/3201463208 >> v1:172.21.15.170:6816/3298994444 conn(0x7fc92406dfe0 legacy=0x7fc9240704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:30.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.527+0000 7fc93fcba640 10 monclient: shutdown 2024-03-29T11:16:30.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.527+0000 7fc93fcba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:30.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.527+0000 7fc93fcba640 1 -- 172.21.15.170:0/3201463208 >> v1:172.21.15.184:6789/0 conn(0x7fc9380ac5a0 legacy=0x7fc938144ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:30.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.527+0000 7fc93fcba640 1 -- 172.21.15.170:0/3201463208 shutdown_connections 2024-03-29T11:16:30.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.527+0000 7fc93fcba640 1 -- 172.21.15.170:0/3201463208 >> 172.21.15.170:0/3201463208 conn(0x7fc93806b660 msgr2=0x7fc9380b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:30.506 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.527+0000 7fc93fcba640 1 -- 172.21.15.170:0/3201463208 shutdown_connections 2024-03-29T11:16:30.506 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.527+0000 7fc93fcba640 1 -- 172.21.15.170:0/3201463208 wait complete. 2024-03-29T11:16:30.506 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:30.527+0000 7fc93fcba640 1 librados: shutdown 2024-03-29T11:16:30.520 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:37.327 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:37.481 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.501+0000 7fe22a12b640 10 monclient: get_monmap_and_config 2024-03-29T11:16:37.481 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.501+0000 7fe22a12b640 10 monclient: build_initial_monmap 2024-03-29T11:16:37.481 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.501+0000 7fe22a12b640 10 monclient: monmap: 2024-03-29T11:16:37.481 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:16:37.481 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:37.482 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:16:37.503031+0000 2024-03-29T11:16:37.482 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:16:37.503031+0000 2024-03-29T11:16:37.482 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:37.482 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:37.482 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:16:37.483 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:16:37.483 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:16:37.483 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:37.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.502+0000 7fe22a12b640 1 Processor -- start 2024-03-29T11:16:37.484 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.503+0000 7fe22a12b640 1 -- start start 2024-03-29T11:16:37.484 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.503+0000 7fe22a12b640 10 monclient: init 2024-03-29T11:16:37.496 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.504+0000 7fe22a12b640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:37.497 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.504+0000 7fe22a12b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:16:37.497 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.504+0000 7fe22a12b640 10 monclient(hunting): picked mon.noname-b con 0x7fe2240ae250 addr v1:172.21.15.170:6790/0 2024-03-29T11:16:37.497 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.504+0000 7fe22a12b640 10 monclient(hunting): picked mon.noname-c con 0x7fe22413f970 addr v1:172.21.15.184:6789/0 2024-03-29T11:16:37.497 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.504+0000 7fe22a12b640 10 monclient(hunting): picked mon.noname-a con 0x7fe224142800 addr v1:172.21.15.170:6789/0 2024-03-29T11:16:37.497 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.504+0000 7fe22a12b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2240afe00 con 0x7fe224142800 2024-03-29T11:16:37.498 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.504+0000 7fe22a12b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe224142c10 con 0x7fe22413f970 2024-03-29T11:16:37.498 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.504+0000 7fe22a12b640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2240afb20 con 0x7fe2240ae250 2024-03-29T11:16:37.498 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.504+0000 7fe22a12b640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:37.498 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.504+0000 7fe22a12b640 10 monclient(hunting): authenticate will time out at 1178.816304s 2024-03-29T11:16:37.498 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.506+0000 7fe222ffd640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fe22413f970 0x7fe2241420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:60746/0 (socket says 172.21.15.170:60746) 2024-03-29T11:16:37.498 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.506+0000 7fe222ffd640 1 -- 172.21.15.170:0/59643999 learned_addr learned my addr 172.21.15.170:0/59643999 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:16:37.499 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.508+0000 7fe2227fc640 1 -- 172.21.15.170:0/59643999 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037270102 0 0) 0x7fe2240afb20 con 0x7fe2240ae250 2024-03-29T11:16:37.499 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.508+0000 7fe2227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:37.499 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.508+0000 7fe2227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:37.499 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.508+0000 7fe2227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:37.499 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.508+0000 7fe2227fc640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:16:37.499 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.508+0000 7fe2227fc640 1 -- 172.21.15.170:0/59643999 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe20c003520 con 0x7fe2240ae250 2024-03-29T11:16:37.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.508+0000 7fe2227fc640 1 -- 172.21.15.170:0/59643999 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1463603749 0 0) 0x7fe20c003520 con 0x7fe2240ae250 2024-03-29T11:16:37.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.508+0000 7fe2227fc640 1 -- 172.21.15.170:0/59643999 >> v1:172.21.15.184:6789/0 conn(0x7fe22413f970 legacy=0x7fe2241420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:37.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.508+0000 7fe2227fc640 1 -- 172.21.15.170:0/59643999 >> v1:172.21.15.170:6789/0 conn(0x7fe224142800 legacy=0x7fe22414da10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:16:37.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe2227fc640 10 monclient: _finish_hunting 0 2024-03-29T11:16:37.500 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe2227fc640 1 monclient: found mon.noname-b 2024-03-29T11:16:37.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe2227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:37.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe2227fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:16:37.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe2227fc640 1 -- 172.21.15.170:0/59643999 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2240aab80 con 0x7fe2240ae250 2024-03-29T11:16:37.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe2227fc640 10 monclient: _finish_auth 0 2024-03-29T11:16:37.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe2227fc640 10 monclient: _check_auth_tickets 2024-03-29T11:16:37.501 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe2227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:37.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe2227fc640 1 -- 172.21.15.170:0/59643999 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe214001670 con 0x7fe2240ae250 2024-03-29T11:16:37.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe2227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:37.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe2227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:37.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe2227fc640 10 monclient: dump: 2024-03-29T11:16:37.502 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:37.502 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:37.503 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:37.503 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:37.503 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:37.503 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:37.503 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:37.504 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:37.504 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:37.504 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:37.504 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe22a12b640 5 monclient: authenticate success, global_id 4415 2024-03-29T11:16:37.504 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe22a12b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:37.504 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe2227fc640 1 -- 172.21.15.170:0/59643999 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe214001900 con 0x7fe2240ae250 2024-03-29T11:16:37.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe2227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:37.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe2227fc640 1 -- 172.21.15.170:0/59643999 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe214001cd0 con 0x7fe2240ae250 2024-03-29T11:16:37.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe2227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:37.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe2227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:37.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe2227fc640 10 monclient: dump: 2024-03-29T11:16:37.505 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:37.506 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:37.506 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:37.506 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:37.506 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:37.506 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:37.506 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:37.507 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:37.507 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:37.507 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:37.507 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe22a12b640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:37.507 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.509+0000 7fe22a12b640 4 set_mon_vals no callback set 2024-03-29T11:16:37.508 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.510+0000 7fe22a12b640 10 monclient: shutdown 2024-03-29T11:16:37.508 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.511+0000 7fe22a12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:37.508 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.511+0000 7fe22a12b640 1 -- 172.21.15.170:0/59643999 >> v1:172.21.15.170:6790/0 conn(0x7fe2240ae250 legacy=0x7fe22413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:37.508 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.511+0000 7fe22a12b640 1 -- 172.21.15.170:0/59643999 shutdown_connections 2024-03-29T11:16:37.508 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.511+0000 7fe22a12b640 1 -- 172.21.15.170:0/59643999 >> 172.21.15.170:0/59643999 conn(0x7fe2240b35e0 msgr2=0x7fe2240b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:37.508 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.512+0000 7fe22a12b640 1 -- 172.21.15.170:0/59643999 shutdown_connections 2024-03-29T11:16:37.509 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.512+0000 7fe22a12b640 1 -- 172.21.15.170:0/59643999 wait complete. 2024-03-29T11:16:37.509 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.513+0000 7fe22a12b640 10 monclient: build_initial_monmap 2024-03-29T11:16:37.509 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.514+0000 7fe22a12b640 10 monclient: monmap: 2024-03-29T11:16:37.509 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:16:37.509 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:37.509 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:16:37.510 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:16:37.510 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:37.510 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:37.510 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:16:37.510 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:16:37.510 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:16:37.511 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:37.511 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.514+0000 7fe22a12b640 1 librados: starting msgr at 2024-03-29T11:16:37.511 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.514+0000 7fe22a12b640 1 librados: starting objecter 2024-03-29T11:16:37.511 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.514+0000 7fe22a12b640 1 Processor -- start 2024-03-29T11:16:37.511 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.515+0000 7fe22a12b640 1 -- start start 2024-03-29T11:16:37.512 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.515+0000 7fe22a12b640 1 librados: setting wanted keys 2024-03-29T11:16:37.512 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.515+0000 7fe22a12b640 1 librados: calling monclient init 2024-03-29T11:16:37.512 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.515+0000 7fe22a12b640 10 monclient: init 2024-03-29T11:16:37.512 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.516+0000 7fe22a12b640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:37.512 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.516+0000 7fe22a12b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:37.512 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.516+0000 7fe22a12b640 10 monclient(hunting): picked mon.noname-b con 0x7fe2240ae250 addr v1:172.21.15.184:6789/0 2024-03-29T11:16:37.513 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.517+0000 7fe22a12b640 10 monclient(hunting): picked mon.noname-a con 0x7fe22413f970 addr v1:172.21.15.170:6789/0 2024-03-29T11:16:37.513 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.517+0000 7fe22a12b640 10 monclient(hunting): picked mon.noname-c con 0x7fe224142800 addr v1:172.21.15.170:6790/0 2024-03-29T11:16:37.513 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.517+0000 7fe22a12b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2241387f0 con 0x7fe22413f970 2024-03-29T11:16:37.513 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.517+0000 7fe22a12b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22413a830 con 0x7fe2240ae250 2024-03-29T11:16:37.513 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.517+0000 7fe22a12b640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22413a9c0 con 0x7fe224142800 2024-03-29T11:16:37.513 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.517+0000 7fe22a12b640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:37.513 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.517+0000 7fe22a12b640 10 monclient(hunting): authenticate will time out at 1178.829263s 2024-03-29T11:16:37.521 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.520+0000 7fe2237fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fe2240ae250 0x7fe224143fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:32898/0 (socket says 172.21.15.170:32898) 2024-03-29T11:16:37.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.520+0000 7fe2237fe640 1 -- 172.21.15.170:0/2798962823 learned_addr learned my addr 172.21.15.170:0/2798962823 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:16:37.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.520+0000 7fe20bfff640 10 client.?.objecter ms_handle_connect 0x7fe2240ae250 2024-03-29T11:16:37.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.520+0000 7fe20bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:37.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.523+0000 7fe20bfff640 10 client.?.objecter ms_handle_connect 0x7fe22413f970 2024-03-29T11:16:37.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.523+0000 7fe20bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:37.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.523+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250453665 0 0) 0x7fe2241387f0 con 0x7fe22413f970 2024-03-29T11:16:37.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.523+0000 7fe20bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:37.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.523+0000 7fe20bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:37.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.523+0000 7fe20bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:37.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.523+0000 7fe20bfff640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:16:37.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.523+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe210004eb0 con 0x7fe22413f970 2024-03-29T11:16:37.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.523+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197743494 0 0) 0x7fe22413a830 con 0x7fe2240ae250 2024-03-29T11:16:37.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.523+0000 7fe20bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:37.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.523+0000 7fe20bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:37.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.523+0000 7fe20bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:37.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.523+0000 7fe20bfff640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:16:37.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.523+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2241387f0 con 0x7fe2240ae250 2024-03-29T11:16:37.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 10 client.?.objecter ms_handle_connect 0x7fe224142800 2024-03-29T11:16:37.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:37.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 87095866 0 0) 0x7fe210004eb0 con 0x7fe22413f970 2024-03-29T11:16:37.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe22413a830 con 0x7fe22413f970 2024-03-29T11:16:37.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe2180018a0 con 0x7fe22413f970 2024-03-29T11:16:37.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:37.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:37.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 10 monclient(hunting): dump: 2024-03-29T11:16:37.523 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:37.523 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:37.524 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:37.524 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:37.524 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:37.524 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:37.524 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:37.524 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:37.524 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:37.524 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:37.524 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:37.524 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077018952 0 0) 0x7fe21c001370 con 0x7fe224142800 2024-03-29T11:16:37.524 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:37.524 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:37.524 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:37.524 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:16:37.524 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe210004eb0 con 0x7fe224142800 2024-03-29T11:16:37.525 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1862604360 0 0) 0x7fe2241387f0 con 0x7fe2240ae250 2024-03-29T11:16:37.525 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe21c001370 con 0x7fe2240ae250 2024-03-29T11:16:37.525 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe214001cd0 con 0x7fe2240ae250 2024-03-29T11:16:37.525 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:37.525 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:37.525 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.524+0000 7fe20bfff640 10 monclient(hunting): dump: 2024-03-29T11:16:37.525 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:37.525 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:37.525 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:37.525 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:37.525 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:37.525 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:37.525 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:37.525 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:37.526 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:37.526 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:37.526 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4153598962 0 0) 0x7fe21c001370 con 0x7fe2240ae250 2024-03-29T11:16:37.526 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 >> v1:172.21.15.170:6790/0 conn(0x7fe224142800 legacy=0x7fe224071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:37.526 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 >> v1:172.21.15.170:6789/0 conn(0x7fe22413f970 legacy=0x7fe2241d77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:37.526 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe20bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:16:37.526 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe20bfff640 1 monclient: found mon.b 2024-03-29T11:16:37.526 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe20bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:37.526 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe20bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:16:37.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2241d7ed0 con 0x7fe2240ae250 2024-03-29T11:16:37.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe20bfff640 10 monclient: _finish_auth 0 2024-03-29T11:16:37.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe20bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:16:37.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe20bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:37.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe22a12b640 5 monclient: authenticate success, global_id 4393 2024-03-29T11:16:37.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe22a12b640 10 monclient: _renew_subs 2024-03-29T11:16:37.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe22a12b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:16:37.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe22a12b640 1 -- 172.21.15.170:0/2798962823 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2241d8120 con 0x7fe2240ae250 2024-03-29T11:16:37.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe22a12b640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:37.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe22a12b640 10 monclient: _renew_subs 2024-03-29T11:16:37.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe22a12b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:16:37.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe22a12b640 1 -- 172.21.15.170:0/2798962823 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2240724a0 con 0x7fe2240ae250 2024-03-29T11:16:37.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe22a12b640 1 librados: init done 2024-03-29T11:16:37.528 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe214003960 con 0x7fe2240ae250 2024-03-29T11:16:37.528 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe20bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:37.528 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe214003cb0 con 0x7fe2240ae250 2024-03-29T11:16:37.528 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe20bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:37.528 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe20bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:37.528 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe20bfff640 10 monclient: dump: 2024-03-29T11:16:37.528 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:37.528 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:37.528 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:37.529 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:37.529 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:37.529 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:37.529 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:37.529 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:37.529 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:37.529 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:37.530 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.525+0000 7fe221ffb640 4 set_mon_vals no callback set 2024-03-29T11:16:37.530 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.527+0000 7fe22a12b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:37.530 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.527+0000 7fe22a12b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:37.530 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.527+0000 7fe22a12b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:16:37.530 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.527+0000 7fe22a12b640 1 -- 172.21.15.170:0/2798962823 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe204005080 con 0x7fe2240ae250 2024-03-29T11:16:37.530 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.527+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fe214003e10 con 0x7fe2240ae250 2024-03-29T11:16:37.531 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.528+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fe214084ba0 con 0x7fe2240ae250 2024-03-29T11:16:37.531 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.528+0000 7fe20bfff640 10 client.4393.objecter ms_dispatch 0x7fe224000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:37.531 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.528+0000 7fe20bfff640 3 client.4393.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:37.531 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.528+0000 7fe20bfff640 3 client.4393.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:37.531 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.528+0000 7fe20bfff640 20 client.4393.objecter dump_active .. 0 homeless 2024-03-29T11:16:37.531 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.529+0000 7fe222ffd640 10 monclient: get_auth_request con 0x7fe21006dfb0 auth_method 0 2024-03-29T11:16:37.531 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.529+0000 7fe20bfff640 10 client.4393.objecter ms_handle_connect 0x7fe21006dfb0 2024-03-29T11:16:37.532 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.536+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe214050480 con 0x7fe2240ae250 2024-03-29T11:16:37.532 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.536+0000 7fe20bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:37.532 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.536+0000 7fe20bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:37.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.860+0000 7fe22a12b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:37.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.860+0000 7fe22a12b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:37.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.860+0000 7fe22a12b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:16:37.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.860+0000 7fe22a12b640 1 -- 172.21.15.170:0/2798962823 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe204005330 con 0x7fe2240ae250 2024-03-29T11:16:37.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.861+0000 7fe20bfff640 1 -- 172.21.15.170:0/2798962823 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fe2140523b0 con 0x7fe2240ae250 2024-03-29T11:16:37.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.861+0000 7fe20bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:37.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.861+0000 7fe20bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:37.840 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:16:37.840 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:37.844 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.866+0000 7fe22a12b640 10 librados: watch_flush enter 2024-03-29T11:16:37.844 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.866+0000 7fe22a12b640 10 librados: watch_flush exit 2024-03-29T11:16:37.845 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.866+0000 7fe22a12b640 20 client.4393.objecter shutdown clearing up homeless session... 2024-03-29T11:16:37.845 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.866+0000 7fe22a12b640 10 client.4393.objecter successfully canceled tick 2024-03-29T11:16:37.845 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.867+0000 7fe22a12b640 1 -- 172.21.15.170:0/2798962823 >> v1:172.21.15.170:6816/3298994444 conn(0x7fe21006dfb0 legacy=0x7fe210070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:37.845 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.867+0000 7fe22a12b640 10 monclient: shutdown 2024-03-29T11:16:37.845 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.867+0000 7fe22a12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:37.845 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.867+0000 7fe22a12b640 1 -- 172.21.15.170:0/2798962823 >> v1:172.21.15.184:6789/0 conn(0x7fe2240ae250 legacy=0x7fe224143fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:37.845 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.867+0000 7fe22a12b640 1 -- 172.21.15.170:0/2798962823 shutdown_connections 2024-03-29T11:16:37.845 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.867+0000 7fe22a12b640 1 -- 172.21.15.170:0/2798962823 >> 172.21.15.170:0/2798962823 conn(0x7fe2240b35e0 msgr2=0x7fe224142000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:37.845 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.867+0000 7fe22a12b640 1 -- 172.21.15.170:0/2798962823 shutdown_connections 2024-03-29T11:16:37.847 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.867+0000 7fe22a12b640 1 -- 172.21.15.170:0/2798962823 wait complete. 2024-03-29T11:16:37.847 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:37.867+0000 7fe22a12b640 1 librados: shutdown 2024-03-29T11:16:37.866 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:44.673 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:44.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.827+0000 7f7ec170e640 10 monclient: get_monmap_and_config 2024-03-29T11:16:44.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.827+0000 7f7ec170e640 10 monclient: build_initial_monmap 2024-03-29T11:16:44.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.827+0000 7f7ec170e640 10 monclient: monmap: 2024-03-29T11:16:44.805 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:16:44.805 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:44.805 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:16:44.828814+0000 2024-03-29T11:16:44.805 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:16:44.828814+0000 2024-03-29T11:16:44.805 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:44.805 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:44.805 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:16:44.806 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:16:44.806 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:16:44.806 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:44.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.828+0000 7f7ec170e640 1 Processor -- start 2024-03-29T11:16:44.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.828+0000 7f7ec170e640 1 -- start start 2024-03-29T11:16:44.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.828+0000 7f7ec170e640 10 monclient: init 2024-03-29T11:16:44.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.829+0000 7f7ec170e640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:44.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.829+0000 7f7ec170e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:16:44.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.829+0000 7f7ec170e640 10 monclient(hunting): picked mon.noname-b con 0x7f7ebc1457f0 addr v1:172.21.15.170:6790/0 2024-03-29T11:16:44.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.829+0000 7f7ec170e640 10 monclient(hunting): picked mon.noname-c con 0x7f7ebc13f650 addr v1:172.21.15.184:6789/0 2024-03-29T11:16:44.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.829+0000 7f7ec170e640 10 monclient(hunting): picked mon.noname-a con 0x7f7ebc0a9d70 addr v1:172.21.15.170:6789/0 2024-03-29T11:16:44.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.829+0000 7f7ec170e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ebc0aaef0 con 0x7f7ebc0a9d70 2024-03-29T11:16:44.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.829+0000 7f7ec170e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ebc145bd0 con 0x7f7ebc13f650 2024-03-29T11:16:44.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.829+0000 7f7ec170e640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ebc145eb0 con 0x7f7ebc1457f0 2024-03-29T11:16:44.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.829+0000 7f7ec170e640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:44.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.829+0000 7f7ec170e640 10 monclient(hunting): authenticate will time out at 1186.141777s 2024-03-29T11:16:44.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.829+0000 7f7eba7fc640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f7ebc13f650 0x7f7ebc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:35170/0 (socket says 172.21.15.170:35170) 2024-03-29T11:16:44.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.829+0000 7f7eba7fc640 1 -- 172.21.15.170:0/4084625563 learned_addr learned my addr 172.21.15.170:0/4084625563 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:16:44.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.830+0000 7f7eb9ffb640 1 -- 172.21.15.170:0/4084625563 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494885987 0 0) 0x7f7ebc145bd0 con 0x7f7ebc13f650 2024-03-29T11:16:44.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.830+0000 7f7eb9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:44.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.830+0000 7f7eb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:44.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.830+0000 7f7eb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:44.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.830+0000 7f7eb9ffb640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:16:44.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.830+0000 7f7eb9ffb640 1 -- 172.21.15.170:0/4084625563 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea4003520 con 0x7f7ebc13f650 2024-03-29T11:16:44.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.830+0000 7f7eb9ffb640 1 -- 172.21.15.170:0/4084625563 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271380499 0 0) 0x7f7ebc145eb0 con 0x7f7ebc1457f0 2024-03-29T11:16:44.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.830+0000 7f7eb9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:44.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.830+0000 7f7eb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:44.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.830+0000 7f7eb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:44.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.830+0000 7f7eb9ffb640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:16:44.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.830+0000 7f7eb9ffb640 1 -- 172.21.15.170:0/4084625563 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea4003840 con 0x7f7ebc1457f0 2024-03-29T11:16:44.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.830+0000 7f7eb9ffb640 1 -- 172.21.15.170:0/4084625563 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 80119796 0 0) 0x7f7ea4003520 con 0x7f7ebc13f650 2024-03-29T11:16:44.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.830+0000 7f7eb9ffb640 1 -- 172.21.15.170:0/4084625563 >> v1:172.21.15.170:6790/0 conn(0x7f7ebc1457f0 legacy=0x7f7ebc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:44.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.830+0000 7f7eb9ffb640 1 -- 172.21.15.170:0/4084625563 >> v1:172.21.15.170:6789/0 conn(0x7f7ebc0a9d70 legacy=0x7f7ebc14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:16:44.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.832+0000 7f7eb9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:16:44.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.832+0000 7f7eb9ffb640 1 monclient: found mon.noname-c 2024-03-29T11:16:44.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.832+0000 7f7eb9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:44.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.832+0000 7f7eb9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:16:44.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.832+0000 7f7eb9ffb640 1 -- 172.21.15.170:0/4084625563 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc146340 con 0x7f7ebc13f650 2024-03-29T11:16:44.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.832+0000 7f7eb9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:16:44.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.832+0000 7f7eb9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:16:44.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.832+0000 7f7eb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:44.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.832+0000 7f7eb9ffb640 1 -- 172.21.15.170:0/4084625563 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f7eb0001670 con 0x7f7ebc13f650 2024-03-29T11:16:44.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.832+0000 7f7eb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:44.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.832+0000 7f7eb9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:44.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.832+0000 7f7eb9ffb640 10 monclient: dump: 2024-03-29T11:16:44.828 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:44.829 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:44.829 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:44.829 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:44.829 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:44.829 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:44.829 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:44.829 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:44.829 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:44.829 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:44.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.832+0000 7f7ec170e640 5 monclient: authenticate success, global_id 4396 2024-03-29T11:16:44.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.832+0000 7f7ec170e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:44.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.833+0000 7f7eb9ffb640 1 -- 172.21.15.170:0/4084625563 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7eb0001850 con 0x7f7ebc13f650 2024-03-29T11:16:44.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.833+0000 7f7eb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:44.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.833+0000 7f7eb9ffb640 1 -- 172.21.15.170:0/4084625563 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f7eb0001c60 con 0x7f7ebc13f650 2024-03-29T11:16:44.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.833+0000 7f7eb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:44.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.833+0000 7f7eb9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:44.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.833+0000 7f7eb9ffb640 10 monclient: dump: 2024-03-29T11:16:44.830 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:44.830 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:44.830 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:44.830 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:44.830 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:44.830 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:44.830 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:44.830 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:44.830 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:44.830 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:44.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.833+0000 7f7ec170e640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:44.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.833+0000 7f7ec170e640 4 set_mon_vals no callback set 2024-03-29T11:16:44.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.834+0000 7f7ec170e640 10 monclient: shutdown 2024-03-29T11:16:44.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.834+0000 7f7ec170e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:44.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.834+0000 7f7ec170e640 1 -- 172.21.15.170:0/4084625563 >> v1:172.21.15.184:6789/0 conn(0x7f7ebc13f650 legacy=0x7f7ebc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:44.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.834+0000 7f7ec170e640 1 -- 172.21.15.170:0/4084625563 shutdown_connections 2024-03-29T11:16:44.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.834+0000 7f7ec170e640 1 -- 172.21.15.170:0/4084625563 >> 172.21.15.170:0/4084625563 conn(0x7f7ebc066a00 msgr2=0x7f7ebc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:44.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.835+0000 7f7ec170e640 1 -- 172.21.15.170:0/4084625563 shutdown_connections 2024-03-29T11:16:44.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.835+0000 7f7ec170e640 1 -- 172.21.15.170:0/4084625563 wait complete. 2024-03-29T11:16:44.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.836+0000 7f7ec170e640 10 monclient: build_initial_monmap 2024-03-29T11:16:44.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.836+0000 7f7ec170e640 10 monclient: monmap: 2024-03-29T11:16:44.831 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:16:44.831 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:44.831 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:16:44.831 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:16:44.831 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:44.832 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:44.832 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:16:44.832 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:16:44.832 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:16:44.832 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:44.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.836+0000 7f7ec170e640 1 librados: starting msgr at 2024-03-29T11:16:44.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.836+0000 7f7ec170e640 1 librados: starting objecter 2024-03-29T11:16:44.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.837+0000 7f7ec170e640 1 Processor -- start 2024-03-29T11:16:44.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.837+0000 7f7ec170e640 1 -- start start 2024-03-29T11:16:44.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.837+0000 7f7ec170e640 1 librados: setting wanted keys 2024-03-29T11:16:44.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.837+0000 7f7ec170e640 1 librados: calling monclient init 2024-03-29T11:16:44.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.837+0000 7f7ec170e640 10 monclient: init 2024-03-29T11:16:44.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.838+0000 7f7ec170e640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:44.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.838+0000 7f7ec170e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:44.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.838+0000 7f7ec170e640 10 monclient(hunting): picked mon.noname-a con 0x7f7ebc0a9d70 addr v1:172.21.15.170:6789/0 2024-03-29T11:16:44.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.838+0000 7f7ec170e640 10 monclient(hunting): picked mon.noname-b con 0x7f7ebc13f650 addr v1:172.21.15.184:6789/0 2024-03-29T11:16:44.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.838+0000 7f7ec170e640 10 monclient(hunting): picked mon.noname-c con 0x7f7ebc1457f0 addr v1:172.21.15.170:6790/0 2024-03-29T11:16:44.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.838+0000 7f7ec170e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ebc0adf20 con 0x7f7ebc0a9d70 2024-03-29T11:16:44.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.838+0000 7f7ec170e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ebc141e80 con 0x7f7ebc13f650 2024-03-29T11:16:44.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.838+0000 7f7ec170e640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ebc142010 con 0x7f7ebc1457f0 2024-03-29T11:16:44.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.838+0000 7f7ec170e640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:44.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.838+0000 7f7ec170e640 10 monclient(hunting): authenticate will time out at 1186.150541s 2024-03-29T11:16:44.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.838+0000 7f7ebb7fe640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f7ebc1457f0 0x7f7ebc1e1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:50752/0 (socket says 172.21.15.170:50752) 2024-03-29T11:16:44.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.838+0000 7f7ebb7fe640 1 -- 172.21.15.170:0/2716621475 learned_addr learned my addr 172.21.15.170:0/2716621475 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:16:44.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.839+0000 7f7ea37fe640 10 client.?.objecter ms_handle_connect 0x7f7ebc1457f0 2024-03-29T11:16:44.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.839+0000 7f7ea37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:44.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.839+0000 7f7ea37fe640 1 -- 172.21.15.170:0/2716621475 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533716841 0 0) 0x7f7ebc142010 con 0x7f7ebc1457f0 2024-03-29T11:16:44.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.839+0000 7f7ea37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:44.834 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.839+0000 7f7ea37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:44.834 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.839+0000 7f7ea37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:44.834 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.839+0000 7f7ea37fe640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:16:44.834 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.839+0000 7f7ea37fe640 1 -- 172.21.15.170:0/2716621475 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea8004eb0 con 0x7f7ebc1457f0 2024-03-29T11:16:44.834 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.840+0000 7f7ea37fe640 1 -- 172.21.15.170:0/2716621475 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3329064901 0 0) 0x7f7ea8004eb0 con 0x7f7ebc1457f0 2024-03-29T11:16:44.834 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.840+0000 7f7ea37fe640 1 -- 172.21.15.170:0/2716621475 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ebc142010 con 0x7f7ebc1457f0 2024-03-29T11:16:44.834 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.840+0000 7f7ea37fe640 1 -- 172.21.15.170:0/2716621475 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f7eb4004380 con 0x7f7ebc1457f0 2024-03-29T11:16:44.834 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.840+0000 7f7ea37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:44.834 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.840+0000 7f7ea37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:44.834 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.840+0000 7f7ea37fe640 10 monclient(hunting): dump: 2024-03-29T11:16:44.834 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:44.834 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:44.834 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:44.835 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:44.835 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:44.835 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:44.835 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:44.835 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:44.835 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:44.835 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:44.835 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.840+0000 7f7ea37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:44.835 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.840+0000 7f7ea37fe640 1 -- 172.21.15.170:0/2716621475 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3715707843 0 0) 0x7f7ebc142010 con 0x7f7ebc1457f0 2024-03-29T11:16:44.835 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.840+0000 7f7ea37fe640 1 -- 172.21.15.170:0/2716621475 >> v1:172.21.15.184:6789/0 conn(0x7f7ebc13f650 legacy=0x7f7ebc1d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:44.835 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.840+0000 7f7ea37fe640 1 -- 172.21.15.170:0/2716621475 >> v1:172.21.15.170:6789/0 conn(0x7f7ebc0a9d70 legacy=0x7f7ebc1d7110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:16:44.835 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.841+0000 7f7ea37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:16:44.835 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.841+0000 7f7ea37fe640 1 monclient: found mon.c 2024-03-29T11:16:44.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.841+0000 7f7ea37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:44.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.841+0000 7f7ea37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:16:44.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.841+0000 7f7ea37fe640 1 -- 172.21.15.170:0/2716621475 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc1421a0 con 0x7f7ebc1457f0 2024-03-29T11:16:44.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.841+0000 7f7ea37fe640 10 monclient: _finish_auth 0 2024-03-29T11:16:44.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.841+0000 7f7ea37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:16:44.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.841+0000 7f7ea37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:44.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.841+0000 7f7ec170e640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:16:44.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.841+0000 7f7ec170e640 10 monclient: _renew_subs 2024-03-29T11:16:44.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.841+0000 7f7ec170e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:16:44.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.841+0000 7f7ec170e640 1 -- 172.21.15.170:0/2716621475 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ebc0ad200 con 0x7f7ebc1457f0 2024-03-29T11:16:44.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.841+0000 7f7ec170e640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:44.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.841+0000 7f7ec170e640 10 monclient: _renew_subs 2024-03-29T11:16:44.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.841+0000 7f7ec170e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:16:44.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.841+0000 7f7ec170e640 1 -- 172.21.15.170:0/2716621475 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ebc0ad7c0 con 0x7f7ebc1457f0 2024-03-29T11:16:44.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.841+0000 7f7ec170e640 1 librados: init done 2024-03-29T11:16:44.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.843+0000 7f7ec170e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:44.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.843+0000 7f7ec170e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:44.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.843+0000 7f7ec170e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:16:44.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.843+0000 7f7ec170e640 1 -- 172.21.15.170:0/2716621475 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ebc075f10 con 0x7f7ebc1457f0 2024-03-29T11:16:44.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.843+0000 7f7ea37fe640 1 -- 172.21.15.170:0/2716621475 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7eb40047d0 con 0x7f7ebc1457f0 2024-03-29T11:16:44.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.843+0000 7f7ea37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:44.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.843+0000 7f7ea37fe640 1 -- 172.21.15.170:0/2716621475 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f7eb4001500 con 0x7f7ebc1457f0 2024-03-29T11:16:44.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.843+0000 7f7ea37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:44.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.843+0000 7f7ea37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:44.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.843+0000 7f7ea37fe640 10 monclient: dump: 2024-03-29T11:16:44.837 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:44.837 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:44.837 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:44.838 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:44.838 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:44.838 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:44.838 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:44.838 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:44.838 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:44.838 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:44.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.843+0000 7f7eb97fa640 4 set_mon_vals no callback set 2024-03-29T11:16:44.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.844+0000 7f7ea37fe640 1 -- 172.21.15.170:0/2716621475 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f7eb40016e0 con 0x7f7ebc1457f0 2024-03-29T11:16:44.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.844+0000 7f7ea37fe640 1 -- 172.21.15.170:0/2716621475 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f7eb4084e10 con 0x7f7ebc1457f0 2024-03-29T11:16:44.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.844+0000 7f7ea37fe640 10 client.4424.objecter ms_dispatch 0x7f7ebc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:44.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.844+0000 7f7ea37fe640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:44.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.844+0000 7f7ea37fe640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:44.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.845+0000 7f7ea37fe640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-29T11:16:44.840 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.845+0000 7f7ebaffd640 10 monclient: get_auth_request con 0x7f7ea806dc90 auth_method 0 2024-03-29T11:16:44.840 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.845+0000 7f7ea37fe640 10 client.4424.objecter ms_handle_connect 0x7f7ea806dc90 2024-03-29T11:16:44.840 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.851+0000 7f7ea37fe640 1 -- 172.21.15.170:0/2716621475 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7eb4050680 con 0x7f7ebc1457f0 2024-03-29T11:16:44.841 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.851+0000 7f7ea37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:44.841 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:44.851+0000 7f7ea37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:45.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.152+0000 7f7ec170e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:45.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.152+0000 7f7ec170e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:45.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.152+0000 7f7ec170e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:16:45.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.152+0000 7f7ec170e640 1 -- 172.21.15.170:0/2716621475 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ebc0b3680 con 0x7f7ebc1457f0 2024-03-29T11:16:45.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.153+0000 7f7ea37fe640 1 -- 172.21.15.170:0/2716621475 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f7eb4052780 con 0x7f7ebc1457f0 2024-03-29T11:16:45.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.153+0000 7f7ea37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:45.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.153+0000 7f7ea37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:45.131 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:16:45.131 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:45.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.155+0000 7f7ec170e640 10 librados: watch_flush enter 2024-03-29T11:16:45.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.155+0000 7f7ec170e640 10 librados: watch_flush exit 2024-03-29T11:16:45.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.155+0000 7f7ec170e640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-29T11:16:45.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.155+0000 7f7ec170e640 10 client.4424.objecter successfully canceled tick 2024-03-29T11:16:45.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.155+0000 7f7ec170e640 1 -- 172.21.15.170:0/2716621475 >> v1:172.21.15.170:6816/3298994444 conn(0x7f7ea806dc90 legacy=0x7f7ea8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:45.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.155+0000 7f7ec170e640 10 monclient: shutdown 2024-03-29T11:16:45.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.155+0000 7f7ec170e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:45.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.155+0000 7f7ec170e640 1 -- 172.21.15.170:0/2716621475 >> v1:172.21.15.170:6790/0 conn(0x7f7ebc1457f0 legacy=0x7f7ebc1e1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:45.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.156+0000 7f7ec170e640 1 -- 172.21.15.170:0/2716621475 shutdown_connections 2024-03-29T11:16:45.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.156+0000 7f7ec170e640 1 -- 172.21.15.170:0/2716621475 >> 172.21.15.170:0/2716621475 conn(0x7f7ebc066a00 msgr2=0x7f7ebc13ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:45.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.156+0000 7f7ec170e640 1 -- 172.21.15.170:0/2716621475 shutdown_connections 2024-03-29T11:16:45.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.156+0000 7f7ec170e640 1 -- 172.21.15.170:0/2716621475 wait complete. 2024-03-29T11:16:45.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:45.156+0000 7f7ec170e640 1 librados: shutdown 2024-03-29T11:16:45.149 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:51.956 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.128+0000 7f3ed57a5640 10 monclient: get_monmap_and_config 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.128+0000 7f3ed57a5640 10 monclient: build_initial_monmap 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.128+0000 7f3ed57a5640 10 monclient: monmap: 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:16:52.128793+0000 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:16:52.128793+0000 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:16:52.106 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:16:52.106 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:16:52.106 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:52.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.129+0000 7f3ed57a5640 1 Processor -- start 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.129+0000 7f3ed57a5640 1 -- start start 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.129+0000 7f3ed57a5640 10 monclient: init 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.130+0000 7f3ed57a5640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.130+0000 7f3ed57a5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.130+0000 7f3ed57a5640 10 monclient(hunting): picked mon.noname-c con 0x7f3ec8038a10 addr v1:172.21.15.184:6789/0 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.130+0000 7f3ed57a5640 10 monclient(hunting): picked mon.noname-a con 0x7f3ec8038df0 addr v1:172.21.15.170:6789/0 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.130+0000 7f3ed57a5640 10 monclient(hunting): picked mon.noname-b con 0x7f3ec8030690 addr v1:172.21.15.170:6790/0 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.133+0000 7f3ed57a5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec80c54f0 con 0x7f3ec8038df0 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.133+0000 7f3ed57a5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec8036460 con 0x7f3ec8038a10 2024-03-29T11:16:52.111 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.133+0000 7f3ed57a5640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec8039870 con 0x7f3ec8030690 2024-03-29T11:16:52.111 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.133+0000 7f3ed57a5640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:52.111 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.133+0000 7f3ed57a5640 10 monclient(hunting): authenticate will time out at 1193.444163s 2024-03-29T11:16:52.111 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.132+0000 7f3ecf7fe640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f3ec8030690 0x7f3ec80d31a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:55307/0 (socket says 172.21.15.170:55307) 2024-03-29T11:16:52.111 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.133+0000 7f3ecf7fe640 1 -- 172.21.15.170:0/2682888858 learned_addr learned my addr 172.21.15.170:0/2682888858 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:16:52.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.137+0000 7f3ecdffb640 1 -- 172.21.15.170:0/2682888858 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949288249 0 0) 0x7f3ec80c54f0 con 0x7f3ec8038df0 2024-03-29T11:16:52.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.137+0000 7f3ecdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:52.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.137+0000 7f3ecdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:52.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.137+0000 7f3ecdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:52.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.137+0000 7f3ecdffb640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:16:52.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.137+0000 7f3ecdffb640 1 -- 172.21.15.170:0/2682888858 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eb8003520 con 0x7f3ec8038df0 2024-03-29T11:16:52.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.137+0000 7f3ecdffb640 1 -- 172.21.15.170:0/2682888858 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 947176882 0 0) 0x7f3ec8039870 con 0x7f3ec8030690 2024-03-29T11:16:52.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.137+0000 7f3ecdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:52.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.137+0000 7f3ecdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:52.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.137+0000 7f3ecdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:52.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.137+0000 7f3ecdffb640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:16:52.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.138+0000 7f3ecdffb640 1 -- 172.21.15.170:0/2682888858 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ec80c54f0 con 0x7f3ec8030690 2024-03-29T11:16:52.120 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 1 -- 172.21.15.170:0/2682888858 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2651101787 0 0) 0x7f3eb8003520 con 0x7f3ec8038df0 2024-03-29T11:16:52.120 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 1 -- 172.21.15.170:0/2682888858 >> v1:172.21.15.170:6790/0 conn(0x7f3ec8030690 legacy=0x7f3ec80d31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:52.120 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 1 -- 172.21.15.170:0/2682888858 >> v1:172.21.15.184:6789/0 conn(0x7f3ec8038a10 legacy=0x7f3ec8033550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:52.120 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:16:52.120 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 1 monclient: found mon.noname-a 2024-03-29T11:16:52.120 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:52.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:16:52.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 1 -- 172.21.15.170:0/2682888858 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec80356c0 con 0x7f3ec8038df0 2024-03-29T11:16:52.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 10 monclient: _finish_auth 0 2024-03-29T11:16:52.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:16:52.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:52.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 1 -- 172.21.15.170:0/2682888858 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3ec4001690 con 0x7f3ec8038df0 2024-03-29T11:16:52.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:52.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:52.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 10 monclient: dump: 2024-03-29T11:16:52.121 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:52.122 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:52.122 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:52.122 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:52.122 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:52.122 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:52.122 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:52.122 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:52.122 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:52.122 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:52.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ed57a5640 5 monclient: authenticate success, global_id 4404 2024-03-29T11:16:52.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ed57a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:52.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 1 -- 172.21.15.170:0/2682888858 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ec4001860 con 0x7f3ec8038df0 2024-03-29T11:16:52.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:52.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 1 -- 172.21.15.170:0/2682888858 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3ec4001cb0 con 0x7f3ec8038df0 2024-03-29T11:16:52.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:52.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:52.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ecdffb640 10 monclient: dump: 2024-03-29T11:16:52.123 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:52.123 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:52.123 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:52.123 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:52.124 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:52.124 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:52.124 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:52.124 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:52.124 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:52.124 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:52.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ed57a5640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:52.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.139+0000 7f3ed57a5640 4 set_mon_vals no callback set 2024-03-29T11:16:52.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.140+0000 7f3ed57a5640 10 monclient: shutdown 2024-03-29T11:16:52.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.140+0000 7f3ed57a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:52.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.140+0000 7f3ed57a5640 1 -- 172.21.15.170:0/2682888858 >> v1:172.21.15.170:6789/0 conn(0x7f3ec8038df0 legacy=0x7f3ec80cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:52.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.141+0000 7f3ed57a5640 1 -- 172.21.15.170:0/2682888858 shutdown_connections 2024-03-29T11:16:52.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.141+0000 7f3ed57a5640 1 -- 172.21.15.170:0/2682888858 >> 172.21.15.170:0/2682888858 conn(0x7f3ec8005c30 msgr2=0x7f3ec8006040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:52.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.141+0000 7f3ed57a5640 1 -- 172.21.15.170:0/2682888858 shutdown_connections 2024-03-29T11:16:52.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.141+0000 7f3ed57a5640 1 -- 172.21.15.170:0/2682888858 wait complete. 2024-03-29T11:16:52.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.142+0000 7f3ed57a5640 10 monclient: build_initial_monmap 2024-03-29T11:16:52.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.142+0000 7f3ed57a5640 10 monclient: monmap: 2024-03-29T11:16:52.125 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:16:52.125 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:52.125 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:16:52.125 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:16:52.126 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:52.126 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:52.126 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:16:52.126 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:16:52.126 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:16:52.126 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:52.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.143+0000 7f3ed57a5640 1 librados: starting msgr at 2024-03-29T11:16:52.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.143+0000 7f3ed57a5640 1 librados: starting objecter 2024-03-29T11:16:52.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.143+0000 7f3ed57a5640 1 Processor -- start 2024-03-29T11:16:52.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.143+0000 7f3ed57a5640 1 -- start start 2024-03-29T11:16:52.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.143+0000 7f3ed57a5640 1 librados: setting wanted keys 2024-03-29T11:16:52.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.143+0000 7f3ed57a5640 1 librados: calling monclient init 2024-03-29T11:16:52.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.144+0000 7f3ed57a5640 10 monclient: init 2024-03-29T11:16:52.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.144+0000 7f3ed57a5640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:52.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.144+0000 7f3ed57a5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:52.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.144+0000 7f3ed57a5640 10 monclient(hunting): picked mon.noname-a con 0x7f3ec8030690 addr v1:172.21.15.170:6789/0 2024-03-29T11:16:52.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.145+0000 7f3ed57a5640 10 monclient(hunting): picked mon.noname-b con 0x7f3ec8038a10 addr v1:172.21.15.184:6789/0 2024-03-29T11:16:52.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.145+0000 7f3ed57a5640 10 monclient(hunting): picked mon.noname-c con 0x7f3ec80ca140 addr v1:172.21.15.170:6790/0 2024-03-29T11:16:52.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.145+0000 7f3ed57a5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec802ea10 con 0x7f3ec8030690 2024-03-29T11:16:52.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.145+0000 7f3ed57a5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec80b3dc0 con 0x7f3ec8038a10 2024-03-29T11:16:52.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.145+0000 7f3ed57a5640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec802d390 con 0x7f3ec80ca140 2024-03-29T11:16:52.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.145+0000 7f3ed57a5640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:52.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.145+0000 7f3ed57a5640 10 monclient(hunting): authenticate will time out at 1193.456471s 2024-03-29T11:16:52.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.148+0000 7f3ece7fc640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f3ec8038a10 0x7f3ec80b2fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:47261/0 (socket says 172.21.15.170:47261) 2024-03-29T11:16:52.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.148+0000 7f3ece7fc640 1 -- 172.21.15.170:0/2122451059 learned_addr learned my addr 172.21.15.170:0/2122451059 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:16:52.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.148+0000 7f3eb77fe640 10 client.?.objecter ms_handle_connect 0x7f3ec8030690 2024-03-29T11:16:52.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.148+0000 7f3eb77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:52.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.149+0000 7f3eb77fe640 10 client.?.objecter ms_handle_connect 0x7f3ec8038a10 2024-03-29T11:16:52.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.149+0000 7f3eb77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:52.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.149+0000 7f3eb77fe640 1 -- 172.21.15.170:0/2122451059 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006850636 0 0) 0x7f3ec802ea10 con 0x7f3ec8030690 2024-03-29T11:16:52.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.149+0000 7f3eb77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:52.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.149+0000 7f3eb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:52.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.149+0000 7f3eb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:52.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.149+0000 7f3eb77fe640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:16:52.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.149+0000 7f3eb77fe640 1 -- 172.21.15.170:0/2122451059 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ebc004eb0 con 0x7f3ec8030690 2024-03-29T11:16:52.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.150+0000 7f3eb77fe640 1 -- 172.21.15.170:0/2122451059 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3522628239 0 0) 0x7f3ebc004eb0 con 0x7f3ec8030690 2024-03-29T11:16:52.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.150+0000 7f3eb77fe640 1 -- 172.21.15.170:0/2122451059 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ec802ea10 con 0x7f3ec8030690 2024-03-29T11:16:52.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.150+0000 7f3eb77fe640 1 -- 172.21.15.170:0/2122451059 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3ec0001870 con 0x7f3ec8030690 2024-03-29T11:16:52.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.150+0000 7f3eb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:52.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.150+0000 7f3eb77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:52.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.150+0000 7f3eb77fe640 10 monclient(hunting): dump: 2024-03-29T11:16:52.132 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:52.132 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:52.132 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:52.132 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:52.132 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:52.132 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:52.132 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:52.132 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:52.133 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:52.133 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:52.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.150+0000 7f3eb77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:52.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.150+0000 7f3eb77fe640 1 -- 172.21.15.170:0/2122451059 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2361171029 0 0) 0x7f3ec802ea10 con 0x7f3ec8030690 2024-03-29T11:16:52.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.150+0000 7f3eb77fe640 1 -- 172.21.15.170:0/2122451059 >> v1:172.21.15.170:6790/0 conn(0x7f3ec80ca140 legacy=0x7f3ec80b36b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:16:52.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.152+0000 7f3eb77fe640 1 -- 172.21.15.170:0/2122451059 >> v1:172.21.15.184:6789/0 conn(0x7f3ec8038a10 legacy=0x7f3ec80b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:52.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.152+0000 7f3eb77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:16:52.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.152+0000 7f3eb77fe640 1 monclient: found mon.a 2024-03-29T11:16:52.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.152+0000 7f3eb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:52.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.152+0000 7f3eb77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:52.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.152+0000 7f3eb77fe640 1 -- 172.21.15.170:0/2122451059 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec80cace0 con 0x7f3ec8030690 2024-03-29T11:16:52.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.152+0000 7f3eb77fe640 10 monclient: _finish_auth 0 2024-03-29T11:16:52.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.152+0000 7f3eb77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:16:52.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.152+0000 7f3eb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:52.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.152+0000 7f3ed57a5640 5 monclient: authenticate success, global_id 4407 2024-03-29T11:16:52.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.152+0000 7f3ed57a5640 10 monclient: _renew_subs 2024-03-29T11:16:52.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.152+0000 7f3ed57a5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:52.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.152+0000 7f3ed57a5640 1 -- 172.21.15.170:0/2122451059 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec80b6640 con 0x7f3ec8030690 2024-03-29T11:16:52.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.152+0000 7f3ed57a5640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:52.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.152+0000 7f3ed57a5640 10 monclient: _renew_subs 2024-03-29T11:16:52.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.152+0000 7f3ed57a5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:52.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.153+0000 7f3ed57a5640 1 -- 172.21.15.170:0/2122451059 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec8164a70 con 0x7f3ec8030690 2024-03-29T11:16:52.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.153+0000 7f3ed57a5640 1 librados: init done 2024-03-29T11:16:52.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.155+0000 7f3eb57fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:52.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.155+0000 7f3eb57fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:52.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.155+0000 7f3eb57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:52.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.155+0000 7f3eb77fe640 1 -- 172.21.15.170:0/2122451059 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ec00035f0 con 0x7f3ec8030690 2024-03-29T11:16:52.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.155+0000 7f3eb57fa640 1 -- 172.21.15.170:0/2122451059 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec8007bc0 con 0x7f3ec8030690 2024-03-29T11:16:52.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.155+0000 7f3eb77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:52.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.155+0000 7f3eb77fe640 1 -- 172.21.15.170:0/2122451059 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3ec0003900 con 0x7f3ec8030690 2024-03-29T11:16:52.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.155+0000 7f3eb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:52.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.155+0000 7f3eb77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:52.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.155+0000 7f3eb77fe640 10 monclient: dump: 2024-03-29T11:16:52.136 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:52.136 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:52.136 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:52.136 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:52.136 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:52.136 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:52.136 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:52.136 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:52.136 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:52.136 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:52.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.155+0000 7f3ecd7fa640 4 set_mon_vals no callback set 2024-03-29T11:16:52.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.157+0000 7f3eb77fe640 1 -- 172.21.15.170:0/2122451059 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f3ec00198d0 con 0x7f3ec8030690 2024-03-29T11:16:52.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.158+0000 7f3eb77fe640 1 -- 172.21.15.170:0/2122451059 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f3ec00837e0 con 0x7f3ec8030690 2024-03-29T11:16:52.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.158+0000 7f3eb77fe640 10 client.4407.objecter ms_dispatch 0x7f3ed0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:52.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.158+0000 7f3eb77fe640 3 client.4407.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:52.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.158+0000 7f3eb77fe640 3 client.4407.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:52.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.158+0000 7f3eb77fe640 20 client.4407.objecter dump_active .. 0 homeless 2024-03-29T11:16:52.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.160+0000 7f3ece7fc640 10 monclient: get_auth_request con 0x7f3ebc06dc60 auth_method 0 2024-03-29T11:16:52.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.161+0000 7f3eb77fe640 10 client.4407.objecter ms_handle_connect 0x7f3ebc06dc60 2024-03-29T11:16:52.142 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.165+0000 7f3eb77fe640 1 -- 172.21.15.170:0/2122451059 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3ec004eef0 con 0x7f3ec8030690 2024-03-29T11:16:52.142 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.166+0000 7f3eb77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:52.142 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.166+0000 7f3eb77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:52.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.492+0000 7f3eb57fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:52.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.492+0000 7f3eb57fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:52.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.492+0000 7f3eb57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:52.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.492+0000 7f3eb57fa640 1 -- 172.21.15.170:0/2122451059 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ec80cae70 con 0x7f3ec8030690 2024-03-29T11:16:52.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.493+0000 7f3eb77fe640 1 -- 172.21.15.170:0/2122451059 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f3ec0050ff0 con 0x7f3ec8030690 2024-03-29T11:16:52.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.493+0000 7f3eb77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:52.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.493+0000 7f3eb77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:52.470 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:16:52.470 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:52.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.497+0000 7f3eb57fa640 10 librados: watch_flush enter 2024-03-29T11:16:52.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.499+0000 7f3eb57fa640 10 librados: watch_flush exit 2024-03-29T11:16:52.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.499+0000 7f3eb57fa640 20 client.4407.objecter shutdown clearing up homeless session... 2024-03-29T11:16:52.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.499+0000 7f3eb57fa640 10 client.4407.objecter successfully canceled tick 2024-03-29T11:16:52.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.499+0000 7f3eb57fa640 1 -- 172.21.15.170:0/2122451059 >> v1:172.21.15.170:6816/3298994444 conn(0x7f3ebc06dc60 legacy=0x7f3ebc070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:52.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.499+0000 7f3eb57fa640 10 monclient: shutdown 2024-03-29T11:16:52.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.499+0000 7f3eb57fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:52.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.499+0000 7f3eb57fa640 1 -- 172.21.15.170:0/2122451059 >> v1:172.21.15.170:6789/0 conn(0x7f3ec8030690 legacy=0x7f3ec8164180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:52.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.499+0000 7f3eb57fa640 1 -- 172.21.15.170:0/2122451059 shutdown_connections 2024-03-29T11:16:52.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.499+0000 7f3eb57fa640 1 -- 172.21.15.170:0/2122451059 >> 172.21.15.170:0/2122451059 conn(0x7f3ec8005c30 msgr2=0x7f3ec8034950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:52.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.500+0000 7f3eb57fa640 1 -- 172.21.15.170:0/2122451059 shutdown_connections 2024-03-29T11:16:52.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.500+0000 7f3eb57fa640 1 -- 172.21.15.170:0/2122451059 wait complete. 2024-03-29T11:16:52.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:52.500+0000 7f3eb57fa640 1 librados: shutdown 2024-03-29T11:16:52.494 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:59.301 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:59.463 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.485+0000 7f5c9cb20640 10 monclient: get_monmap_and_config 2024-03-29T11:16:59.463 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.485+0000 7f5c9cb20640 10 monclient: build_initial_monmap 2024-03-29T11:16:59.463 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.485+0000 7f5c9cb20640 10 monclient: monmap: 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:16:59.486003+0000 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:16:59.486003+0000 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.487+0000 7f5c9cb20640 1 Processor -- start 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.487+0000 7f5c9cb20640 1 -- start start 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.487+0000 7f5c9cb20640 10 monclient: init 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.487+0000 7f5c9cb20640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.487+0000 7f5c9cb20640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.488+0000 7f5c9cb20640 10 monclient(hunting): picked mon.noname-c con 0x7f5c90034850 addr v1:172.21.15.184:6789/0 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.488+0000 7f5c9cb20640 10 monclient(hunting): picked mon.noname-a con 0x7f5c9002a640 addr v1:172.21.15.170:6789/0 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.488+0000 7f5c9cb20640 10 monclient(hunting): picked mon.noname-b con 0x7f5c900327d0 addr v1:172.21.15.170:6790/0 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.488+0000 7f5c9cb20640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9002c820 con 0x7f5c9002a640 2024-03-29T11:16:59.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.488+0000 7f5c9cb20640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9002a120 con 0x7f5c90034850 2024-03-29T11:16:59.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.488+0000 7f5c9cb20640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c900c75b0 con 0x7f5c900327d0 2024-03-29T11:16:59.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.488+0000 7f5c9cb20640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:59.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.488+0000 7f5c9cb20640 10 monclient(hunting): authenticate will time out at 1200.799094s 2024-03-29T11:16:59.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.493+0000 7f5c97fff640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f5c900327d0 0x7f5c900cf240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:48801/0 (socket says 172.21.15.170:48801) 2024-03-29T11:16:59.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c97fff640 1 -- 172.21.15.170:0/1005397889 learned_addr learned my addr 172.21.15.170:0/1005397889 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:16:59.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c977fe640 1 --1- 172.21.15.170:0/1005397889 >> v1:172.21.15.184:6789/0 conn(0x7f5c90034850 0x7f5c900c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:38755/0 (socket says 172.21.15.170:38755) 2024-03-29T11:16:59.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 1 -- 172.21.15.170:0/1005397889 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2221012226 0 0) 0x7f5c9002a120 con 0x7f5c90034850 2024-03-29T11:16:59.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:59.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:59.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:59.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:16:59.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 1 -- 172.21.15.170:0/1005397889 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c88003520 con 0x7f5c90034850 2024-03-29T11:16:59.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 1 -- 172.21.15.170:0/1005397889 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738135526 0 0) 0x7f5c900c75b0 con 0x7f5c900327d0 2024-03-29T11:16:59.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:59.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:59.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:59.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:16:59.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 1 -- 172.21.15.170:0/1005397889 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c9002a120 con 0x7f5c900327d0 2024-03-29T11:16:59.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 1 -- 172.21.15.170:0/1005397889 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 791072417 0 0) 0x7f5c88003520 con 0x7f5c90034850 2024-03-29T11:16:59.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 1 -- 172.21.15.170:0/1005397889 >> v1:172.21.15.170:6790/0 conn(0x7f5c900327d0 legacy=0x7f5c900cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:59.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 1 -- 172.21.15.170:0/1005397889 >> v1:172.21.15.170:6789/0 conn(0x7f5c9002a640 legacy=0x7f5c900320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:59.476 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 10 monclient: _finish_hunting 0 2024-03-29T11:16:59.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 1 monclient: found mon.noname-c 2024-03-29T11:16:59.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:59.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:16:59.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 1 -- 172.21.15.170:0/1005397889 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c90033110 con 0x7f5c90034850 2024-03-29T11:16:59.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 10 monclient: _finish_auth 0 2024-03-29T11:16:59.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 10 monclient: _check_auth_tickets 2024-03-29T11:16:59.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.494+0000 7f5c967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:59.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.495+0000 7f5c9cb20640 5 monclient: authenticate success, global_id 4402 2024-03-29T11:16:59.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.495+0000 7f5c9cb20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:59.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.495+0000 7f5c967fc640 1 -- 172.21.15.170:0/1005397889 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5c9804f730 con 0x7f5c90034850 2024-03-29T11:16:59.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.495+0000 7f5c967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:59.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.495+0000 7f5c967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:59.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.495+0000 7f5c967fc640 10 monclient: dump: 2024-03-29T11:16:59.478 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:59.478 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:59.478 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:59.478 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:59.478 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:59.478 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:59.478 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:59.479 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:59.479 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:59.479 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:59.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.495+0000 7f5c9cb20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:59.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.495+0000 7f5c967fc640 1 -- 172.21.15.170:0/1005397889 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c980008d0 con 0x7f5c90034850 2024-03-29T11:16:59.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.495+0000 7f5c967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:59.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.495+0000 7f5c9cb20640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:59.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.495+0000 7f5c9cb20640 4 set_mon_vals no callback set 2024-03-29T11:16:59.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.496+0000 7f5c9cb20640 10 monclient: shutdown 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.496+0000 7f5c9cb20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.496+0000 7f5c9cb20640 1 -- 172.21.15.170:0/1005397889 >> v1:172.21.15.184:6789/0 conn(0x7f5c90034850 legacy=0x7f5c900c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.496+0000 7f5c9cb20640 1 -- 172.21.15.170:0/1005397889 shutdown_connections 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.496+0000 7f5c9cb20640 1 -- 172.21.15.170:0/1005397889 >> 172.21.15.170:0/1005397889 conn(0x7f5c900058f0 msgr2=0x7f5c90005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.496+0000 7f5c9cb20640 1 -- 172.21.15.170:0/1005397889 shutdown_connections 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.496+0000 7f5c9cb20640 1 -- 172.21.15.170:0/1005397889 wait complete. 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.497+0000 7f5c9cb20640 10 monclient: build_initial_monmap 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.497+0000 7f5c9cb20640 10 monclient: monmap: 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.497+0000 7f5c9cb20640 1 librados: starting msgr at 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.497+0000 7f5c9cb20640 1 librados: starting objecter 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.497+0000 7f5c9cb20640 1 Processor -- start 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.498+0000 7f5c9cb20640 1 -- start start 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.498+0000 7f5c9cb20640 1 librados: setting wanted keys 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.498+0000 7f5c9cb20640 1 librados: calling monclient init 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.498+0000 7f5c9cb20640 10 monclient: init 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.498+0000 7f5c9cb20640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.498+0000 7f5c9cb20640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.498+0000 7f5c9cb20640 10 monclient(hunting): picked mon.noname-a con 0x7f5c9002a640 addr v1:172.21.15.170:6789/0 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.498+0000 7f5c9cb20640 10 monclient(hunting): picked mon.noname-b con 0x7f5c900327d0 addr v1:172.21.15.184:6789/0 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.499+0000 7f5c9cb20640 10 monclient(hunting): picked mon.noname-c con 0x7f5c90034850 addr v1:172.21.15.170:6790/0 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.499+0000 7f5c9cb20640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c900bf9a0 con 0x7f5c9002a640 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.499+0000 7f5c9cb20640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c900bfc10 con 0x7f5c900327d0 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.499+0000 7f5c9cb20640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c900bfef0 con 0x7f5c90034850 2024-03-29T11:16:59.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.499+0000 7f5c9cb20640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:59.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.499+0000 7f5c9cb20640 10 monclient(hunting): authenticate will time out at 1200.810049s 2024-03-29T11:16:59.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.504+0000 7f5c977fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f5c9002a640 0x7f5c90150250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:37927/0 (socket says 172.21.15.170:37927) 2024-03-29T11:16:59.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.504+0000 7f5c977fe640 1 -- 172.21.15.170:0/3259818815 learned_addr learned my addr 172.21.15.170:0/3259818815 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:16:59.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 10 client.?.objecter ms_handle_connect 0x7f5c9002a640 2024-03-29T11:16:59.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:59.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 10 client.?.objecter ms_handle_connect 0x7f5c90034850 2024-03-29T11:16:59.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:59.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 1 -- 172.21.15.170:0/3259818815 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371542486 0 0) 0x7f5c900bf9a0 con 0x7f5c9002a640 2024-03-29T11:16:59.484 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:59.484 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:59.484 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:59.484 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:16:59.484 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 1 -- 172.21.15.170:0/3259818815 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c7c004eb0 con 0x7f5c9002a640 2024-03-29T11:16:59.484 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 1 -- 172.21.15.170:0/3259818815 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724587320 0 0) 0x7f5c900bfef0 con 0x7f5c90034850 2024-03-29T11:16:59.484 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:59.484 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:59.484 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:59.484 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:16:59.484 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 1 -- 172.21.15.170:0/3259818815 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c900bf9a0 con 0x7f5c90034850 2024-03-29T11:16:59.484 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 1 -- 172.21.15.170:0/3259818815 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2919907524 0 0) 0x7f5c7c004eb0 con 0x7f5c9002a640 2024-03-29T11:16:59.484 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 1 -- 172.21.15.170:0/3259818815 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c900bfef0 con 0x7f5c9002a640 2024-03-29T11:16:59.484 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 1 -- 172.21.15.170:0/3259818815 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5c9804f400 con 0x7f5c9002a640 2024-03-29T11:16:59.484 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:59.484 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:59.485 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 10 monclient(hunting): dump: 2024-03-29T11:16:59.485 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:59.485 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:59.485 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:59.485 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:59.485 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:59.485 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:59.485 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:59.485 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:59.486 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:59.486 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:59.486 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.505+0000 7f5c83fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:59.486 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c83fff640 1 -- 172.21.15.170:0/3259818815 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1023310977 0 0) 0x7f5c900bfef0 con 0x7f5c9002a640 2024-03-29T11:16:59.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c83fff640 1 -- 172.21.15.170:0/3259818815 >> v1:172.21.15.170:6790/0 conn(0x7f5c90034850 legacy=0x7f5c90185340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:59.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c83fff640 1 -- 172.21.15.170:0/3259818815 >> v1:172.21.15.184:6789/0 conn(0x7f5c900327d0 legacy=0x7f5c900c0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:59.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c83fff640 10 monclient: _finish_hunting 0 2024-03-29T11:16:59.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c83fff640 1 monclient: found mon.a 2024-03-29T11:16:59.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:59.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c83fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:59.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c83fff640 1 -- 172.21.15.170:0/3259818815 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c90152c60 con 0x7f5c9002a640 2024-03-29T11:16:59.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c83fff640 10 monclient: _finish_auth 0 2024-03-29T11:16:59.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c83fff640 10 monclient: _check_auth_tickets 2024-03-29T11:16:59.487 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:59.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c9cb20640 5 monclient: authenticate success, global_id 4410 2024-03-29T11:16:59.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c9cb20640 10 monclient: _renew_subs 2024-03-29T11:16:59.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c9cb20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:59.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c9cb20640 1 -- 172.21.15.170:0/3259818815 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c900becd0 con 0x7f5c9002a640 2024-03-29T11:16:59.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c9cb20640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:59.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c9cb20640 10 monclient: _renew_subs 2024-03-29T11:16:59.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c9cb20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:59.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c9cb20640 1 -- 172.21.15.170:0/3259818815 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c90150b40 con 0x7f5c9002a640 2024-03-29T11:16:59.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c9cb20640 1 librados: init done 2024-03-29T11:16:59.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c83fff640 1 -- 172.21.15.170:0/3259818815 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c98075710 con 0x7f5c9002a640 2024-03-29T11:16:59.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c83fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:59.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c83fff640 1 -- 172.21.15.170:0/3259818815 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5c9804f730 con 0x7f5c9002a640 2024-03-29T11:16:59.488 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:59.489 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c83fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:59.489 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.506+0000 7f5c83fff640 10 monclient: dump: 2024-03-29T11:16:59.489 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:16:59.489 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:16:59.489 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:59.489 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:16:59.489 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:59.489 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:16:59.489 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:16:59.489 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:16:59.489 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:16:59.489 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:16:59.489 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.507+0000 7f5c95ffb640 4 set_mon_vals no callback set 2024-03-29T11:16:59.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.509+0000 7f5c9cb20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:59.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.509+0000 7f5c9cb20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:59.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.510+0000 7f5c9cb20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:59.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.510+0000 7f5c9cb20640 1 -- 172.21.15.170:0/3259818815 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c78005080 con 0x7f5c9002a640 2024-03-29T11:16:59.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.513+0000 7f5c83fff640 1 -- 172.21.15.170:0/3259818815 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f5c98067ea0 con 0x7f5c9002a640 2024-03-29T11:16:59.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.514+0000 7f5c83fff640 1 -- 172.21.15.170:0/3259818815 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f5c980ff7d0 con 0x7f5c9002a640 2024-03-29T11:16:59.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.514+0000 7f5c83fff640 10 client.4410.objecter ms_dispatch 0x7f5c98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:59.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.514+0000 7f5c83fff640 3 client.4410.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:59.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.514+0000 7f5c83fff640 3 client.4410.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:59.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.514+0000 7f5c83fff640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-29T11:16:59.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.514+0000 7f5c96ffd640 10 monclient: get_auth_request con 0x7f5c7c06dc20 auth_method 0 2024-03-29T11:16:59.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.515+0000 7f5c83fff640 10 client.4410.objecter ms_handle_connect 0x7f5c7c06dc20 2024-03-29T11:16:59.499 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.522+0000 7f5c83fff640 1 -- 172.21.15.170:0/3259818815 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5c980cb0b0 con 0x7f5c9002a640 2024-03-29T11:16:59.499 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.522+0000 7f5c83fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:59.499 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.522+0000 7f5c83fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:59.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.849+0000 7f5c9cb20640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:59.826 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.849+0000 7f5c9cb20640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:59.826 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.849+0000 7f5c9cb20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:16:59.826 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.849+0000 7f5c9cb20640 1 -- 172.21.15.170:0/3259818815 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c78005330 con 0x7f5c9002a640 2024-03-29T11:16:59.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.850+0000 7f5c83fff640 1 -- 172.21.15.170:0/3259818815 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f5c980ccfe0 con 0x7f5c9002a640 2024-03-29T11:16:59.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.851+0000 7f5c83fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:59.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.851+0000 7f5c83fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:59.829 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:16:59.829 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:59.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.856+0000 7f5c9cb20640 10 librados: watch_flush enter 2024-03-29T11:16:59.834 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.858+0000 7f5c9cb20640 10 librados: watch_flush exit 2024-03-29T11:16:59.835 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.858+0000 7f5c9cb20640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-29T11:16:59.835 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.858+0000 7f5c9cb20640 10 client.4410.objecter successfully canceled tick 2024-03-29T11:16:59.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.860+0000 7f5c9cb20640 1 -- 172.21.15.170:0/3259818815 >> v1:172.21.15.170:6816/3298994444 conn(0x7f5c7c06dc20 legacy=0x7f5c7c0700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:59.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.860+0000 7f5c9cb20640 10 monclient: shutdown 2024-03-29T11:16:59.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.860+0000 7f5c9cb20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:59.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.860+0000 7f5c9cb20640 1 -- 172.21.15.170:0/3259818815 >> v1:172.21.15.170:6789/0 conn(0x7f5c9002a640 legacy=0x7f5c90150250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:59.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.863+0000 7f5c9cb20640 1 -- 172.21.15.170:0/3259818815 shutdown_connections 2024-03-29T11:16:59.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.863+0000 7f5c9cb20640 1 -- 172.21.15.170:0/3259818815 >> 172.21.15.170:0/3259818815 conn(0x7f5c900058f0 msgr2=0x7f5c90029610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:59.840 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.863+0000 7f5c9cb20640 1 -- 172.21.15.170:0/3259818815 shutdown_connections 2024-03-29T11:16:59.840 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.863+0000 7f5c9cb20640 1 -- 172.21.15.170:0/3259818815 wait complete. 2024-03-29T11:16:59.840 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:16:59.863+0000 7f5c9cb20640 1 librados: shutdown 2024-03-29T11:16:59.860 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:06.668 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:06.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.851+0000 7f2f2daf0640 10 monclient: get_monmap_and_config 2024-03-29T11:17:06.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.851+0000 7f2f2daf0640 10 monclient: build_initial_monmap 2024-03-29T11:17:06.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.851+0000 7f2f2daf0640 10 monclient: monmap: 2024-03-29T11:17:06.828 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:17:06.828 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:06.828 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:17:06.851755+0000 2024-03-29T11:17:06.828 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:17:06.851755+0000 2024-03-29T11:17:06.828 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:06.828 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:06.828 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:17:06.828 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:17:06.828 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:17:06.829 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:06.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.855+0000 7f2f2daf0640 1 Processor -- start 2024-03-29T11:17:06.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.855+0000 7f2f2daf0640 1 -- start start 2024-03-29T11:17:06.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.855+0000 7f2f2daf0640 10 monclient: init 2024-03-29T11:17:06.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.856+0000 7f2f2daf0640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:06.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.856+0000 7f2f2daf0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:17:06.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.856+0000 7f2f2daf0640 10 monclient(hunting): picked mon.noname-a con 0x7f2f280b2790 addr v1:172.21.15.170:6789/0 2024-03-29T11:17:06.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.856+0000 7f2f2daf0640 10 monclient(hunting): picked mon.noname-c con 0x7f2f28143c80 addr v1:172.21.15.184:6789/0 2024-03-29T11:17:06.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.856+0000 7f2f2daf0640 10 monclient(hunting): picked mon.noname-b con 0x7f2f28146e30 addr v1:172.21.15.170:6790/0 2024-03-29T11:17:06.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.856+0000 7f2f2daf0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f28146290 con 0x7f2f280b2790 2024-03-29T11:17:06.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.856+0000 7f2f2daf0640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f280b3970 con 0x7f2f28143c80 2024-03-29T11:17:06.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.856+0000 7f2f2daf0640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f2806ee80 con 0x7f2f28146e30 2024-03-29T11:17:06.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.856+0000 7f2f2daf0640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:06.833 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.856+0000 7f2f2daf0640 10 monclient(hunting): authenticate will time out at 1208.167144s 2024-03-29T11:17:06.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.859+0000 7f2f2d2ef640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f2f28146e30 0x7f2f28151bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:34005/0 (socket says 172.21.15.170:34005) 2024-03-29T11:17:06.836 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.860+0000 7f2f2d2ef640 1 -- 172.21.15.170:0/556477771 learned_addr learned my addr 172.21.15.170:0/556477771 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:17:06.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.860+0000 7f2f277fe640 1 -- 172.21.15.170:0/556477771 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743410408 0 0) 0x7f2f2806ee80 con 0x7f2f28146e30 2024-03-29T11:17:06.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.860+0000 7f2f277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:06.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.860+0000 7f2f277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:06.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.860+0000 7f2f277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:06.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.860+0000 7f2f277fe640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:17:06.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.860+0000 7f2f277fe640 1 -- 172.21.15.170:0/556477771 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f14003520 con 0x7f2f28146e30 2024-03-29T11:17:06.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.860+0000 7f2f277fe640 1 -- 172.21.15.170:0/556477771 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178236809 0 0) 0x7f2f280b3970 con 0x7f2f28143c80 2024-03-29T11:17:06.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.860+0000 7f2f277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:06.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.860+0000 7f2f277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:06.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.860+0000 7f2f277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:06.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.860+0000 7f2f277fe640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:17:06.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.860+0000 7f2f277fe640 1 -- 172.21.15.170:0/556477771 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f2806ee80 con 0x7f2f28143c80 2024-03-29T11:17:06.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.860+0000 7f2f277fe640 1 -- 172.21.15.170:0/556477771 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1042949286 0 0) 0x7f2f2806ee80 con 0x7f2f28143c80 2024-03-29T11:17:06.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 1 -- 172.21.15.170:0/556477771 >> v1:172.21.15.170:6790/0 conn(0x7f2f28146e30 legacy=0x7f2f28151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:06.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 1 -- 172.21.15.170:0/556477771 >> v1:172.21.15.170:6789/0 conn(0x7f2f280b2790 legacy=0x7f2f28143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:06.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 10 monclient: _finish_hunting 0 2024-03-29T11:17:06.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 1 monclient: found mon.noname-c 2024-03-29T11:17:06.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:06.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:17:06.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 1 -- 172.21.15.170:0/556477771 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2806f160 con 0x7f2f28143c80 2024-03-29T11:17:06.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 10 monclient: _finish_auth 0 2024-03-29T11:17:06.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 10 monclient: _check_auth_tickets 2024-03-29T11:17:06.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:06.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 1 -- 172.21.15.170:0/556477771 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f2f20001920 con 0x7f2f28143c80 2024-03-29T11:17:06.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:06.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:06.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 10 monclient: dump: 2024-03-29T11:17:06.839 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:06.839 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:06.840 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:06.840 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:06.840 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:06.840 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:06.840 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:06.840 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:06.840 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:06.840 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:06.840 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f2daf0640 5 monclient: authenticate success, global_id 4405 2024-03-29T11:17:06.840 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f2daf0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:06.840 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 1 -- 172.21.15.170:0/556477771 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f20001a80 con 0x7f2f28143c80 2024-03-29T11:17:06.840 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:06.840 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 1 -- 172.21.15.170:0/556477771 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f2f20001e10 con 0x7f2f28143c80 2024-03-29T11:17:06.840 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:06.840 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:06.841 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f277fe640 10 monclient: dump: 2024-03-29T11:17:06.841 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:06.841 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:06.841 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:06.841 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:06.841 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:06.841 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:06.841 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:06.841 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:06.841 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:06.841 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:06.841 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f2daf0640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:06.841 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.861+0000 7f2f2daf0640 4 set_mon_vals no callback set 2024-03-29T11:17:06.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.866+0000 7f2f2daf0640 10 monclient: shutdown 2024-03-29T11:17:06.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.866+0000 7f2f2daf0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:06.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.866+0000 7f2f2daf0640 1 -- 172.21.15.170:0/556477771 >> v1:172.21.15.184:6789/0 conn(0x7f2f28143c80 legacy=0x7f2f28146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:06.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.866+0000 7f2f2daf0640 1 -- 172.21.15.170:0/556477771 shutdown_connections 2024-03-29T11:17:06.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.866+0000 7f2f2daf0640 1 -- 172.21.15.170:0/556477771 >> 172.21.15.170:0/556477771 conn(0x7f2f2806d070 msgr2=0x7f2f28066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:06.844 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.867+0000 7f2f2daf0640 1 -- 172.21.15.170:0/556477771 shutdown_connections 2024-03-29T11:17:06.845 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.867+0000 7f2f2daf0640 1 -- 172.21.15.170:0/556477771 wait complete. 2024-03-29T11:17:06.845 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.868+0000 7f2f2daf0640 10 monclient: build_initial_monmap 2024-03-29T11:17:06.845 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.868+0000 7f2f2daf0640 10 monclient: monmap: 2024-03-29T11:17:06.845 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:17:06.845 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:06.845 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:17:06.845 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:17:06.845 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:06.845 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:06.845 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:17:06.846 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:17:06.846 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:17:06.846 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:06.846 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.868+0000 7f2f2daf0640 1 librados: starting msgr at 2024-03-29T11:17:06.847 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.868+0000 7f2f2daf0640 1 librados: starting objecter 2024-03-29T11:17:06.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.868+0000 7f2f2daf0640 1 Processor -- start 2024-03-29T11:17:06.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.870+0000 7f2f2daf0640 1 -- start start 2024-03-29T11:17:06.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.870+0000 7f2f2daf0640 1 librados: setting wanted keys 2024-03-29T11:17:06.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.870+0000 7f2f2daf0640 1 librados: calling monclient init 2024-03-29T11:17:06.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.870+0000 7f2f2daf0640 10 monclient: init 2024-03-29T11:17:06.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.871+0000 7f2f2daf0640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:06.848 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.871+0000 7f2f2daf0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:17:06.851 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.871+0000 7f2f2daf0640 10 monclient(hunting): picked mon.noname-c con 0x7f2f280b2790 addr v1:172.21.15.170:6790/0 2024-03-29T11:17:06.851 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.871+0000 7f2f2daf0640 10 monclient(hunting): picked mon.noname-b con 0x7f2f28143c80 addr v1:172.21.15.184:6789/0 2024-03-29T11:17:06.851 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.871+0000 7f2f2daf0640 10 monclient(hunting): picked mon.noname-a con 0x7f2f28146e30 addr v1:172.21.15.170:6789/0 2024-03-29T11:17:06.851 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.871+0000 7f2f2daf0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f28147d30 con 0x7f2f28146e30 2024-03-29T11:17:06.851 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.871+0000 7f2f2daf0640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f280b6f00 con 0x7f2f28143c80 2024-03-29T11:17:06.851 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.871+0000 7f2f2daf0640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f280b7090 con 0x7f2f280b2790 2024-03-29T11:17:06.851 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.871+0000 7f2f2daf0640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:06.851 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.871+0000 7f2f2daf0640 10 monclient(hunting): authenticate will time out at 1208.182754s 2024-03-29T11:17:06.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.876+0000 7f2f2caee640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f2f280b2790 0x7f2f281daae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:34109/0 (socket says 172.21.15.170:34109) 2024-03-29T11:17:06.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.876+0000 7f2f2caee640 1 -- 172.21.15.170:0/3640464952 learned_addr learned my addr 172.21.15.170:0/3640464952 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:17:06.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.876+0000 7f2f257fa640 10 client.?.objecter ms_handle_connect 0x7f2f280b2790 2024-03-29T11:17:06.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.876+0000 7f2f257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:06.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.876+0000 7f2f257fa640 1 -- 172.21.15.170:0/3640464952 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192326580 0 0) 0x7f2f280b7090 con 0x7f2f280b2790 2024-03-29T11:17:06.853 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.876+0000 7f2f257fa640 10 monclient(hunting): _negotiate 2024-03-29T11:17:06.854 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.876+0000 7f2f257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:06.854 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.876+0000 7f2f257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:06.854 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.876+0000 7f2f257fa640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:17:06.854 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.876+0000 7f2f257fa640 1 -- 172.21.15.170:0/3640464952 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f08004eb0 con 0x7f2f280b2790 2024-03-29T11:17:06.855 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.877+0000 7f2f257fa640 1 -- 172.21.15.170:0/3640464952 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1002293963 0 0) 0x7f2f08004eb0 con 0x7f2f280b2790 2024-03-29T11:17:06.855 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.877+0000 7f2f257fa640 1 -- 172.21.15.170:0/3640464952 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f280b7090 con 0x7f2f280b2790 2024-03-29T11:17:06.855 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.877+0000 7f2f257fa640 1 -- 172.21.15.170:0/3640464952 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f2f10001a30 con 0x7f2f280b2790 2024-03-29T11:17:06.855 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.877+0000 7f2f257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:06.855 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.877+0000 7f2f257fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:06.855 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.877+0000 7f2f257fa640 10 monclient(hunting): dump: 2024-03-29T11:17:06.855 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:06.856 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:06.856 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:06.856 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:06.856 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:06.856 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:06.856 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:06.856 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:06.856 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:06.856 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:06.856 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.877+0000 7f2f257fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:06.856 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.878+0000 7f2f257fa640 10 client.?.objecter ms_handle_connect 0x7f2f28143c80 2024-03-29T11:17:06.856 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.878+0000 7f2f257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:06.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.880+0000 7f2f257fa640 1 -- 172.21.15.170:0/3640464952 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2416457045 0 0) 0x7f2f280b7090 con 0x7f2f280b2790 2024-03-29T11:17:06.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.880+0000 7f2f257fa640 1 -- 172.21.15.170:0/3640464952 >> v1:172.21.15.184:6789/0 conn(0x7f2f28143c80 legacy=0x7f2f281e3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:06.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.880+0000 7f2f257fa640 1 -- 172.21.15.170:0/3640464952 >> v1:172.21.15.170:6789/0 conn(0x7f2f28146e30 legacy=0x7f2f281e44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:06.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.880+0000 7f2f257fa640 10 monclient: _finish_hunting 0 2024-03-29T11:17:06.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.881+0000 7f2f257fa640 1 monclient: found mon.c 2024-03-29T11:17:06.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.881+0000 7f2f257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:06.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.881+0000 7f2f257fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:06.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.881+0000 7f2f257fa640 1 -- 172.21.15.170:0/3640464952 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f280b7220 con 0x7f2f280b2790 2024-03-29T11:17:06.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.881+0000 7f2f257fa640 10 monclient: _finish_auth 0 2024-03-29T11:17:06.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.881+0000 7f2f257fa640 10 monclient: _check_auth_tickets 2024-03-29T11:17:06.858 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.881+0000 7f2f257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:06.861 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.881+0000 7f2f2daf0640 5 monclient: authenticate success, global_id 4439 2024-03-29T11:17:06.861 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.881+0000 7f2f2daf0640 10 monclient: _renew_subs 2024-03-29T11:17:06.861 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.881+0000 7f2f2daf0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:06.862 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.881+0000 7f2f2daf0640 1 -- 172.21.15.170:0/3640464952 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f280708d0 con 0x7f2f280b2790 2024-03-29T11:17:06.862 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.881+0000 7f2f2daf0640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:06.862 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.881+0000 7f2f2daf0640 10 monclient: _renew_subs 2024-03-29T11:17:06.862 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.881+0000 7f2f2daf0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:06.862 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.881+0000 7f2f2daf0640 1 -- 172.21.15.170:0/3640464952 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f281db1f0 con 0x7f2f280b2790 2024-03-29T11:17:06.862 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.881+0000 7f2f2daf0640 1 librados: init done 2024-03-29T11:17:06.862 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.882+0000 7f2f257fa640 1 -- 172.21.15.170:0/3640464952 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f10001c90 con 0x7f2f280b2790 2024-03-29T11:17:06.862 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.882+0000 7f2f257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:06.862 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.882+0000 7f2f257fa640 1 -- 172.21.15.170:0/3640464952 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f2f10003aa0 con 0x7f2f280b2790 2024-03-29T11:17:06.862 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.882+0000 7f2f257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:06.862 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.882+0000 7f2f257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:06.862 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.882+0000 7f2f257fa640 10 monclient: dump: 2024-03-29T11:17:06.862 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:06.862 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:06.863 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:06.863 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:06.863 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:06.863 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:06.863 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:06.863 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:06.863 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:06.863 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:06.863 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.882+0000 7f2f26ffd640 4 set_mon_vals no callback set 2024-03-29T11:17:06.863 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.883+0000 7f2f2daf0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:06.863 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.884+0000 7f2f257fa640 1 -- 172.21.15.170:0/3640464952 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f2f10003c00 con 0x7f2f280b2790 2024-03-29T11:17:06.863 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.884+0000 7f2f2daf0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:06.864 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.884+0000 7f2f2daf0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:06.864 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.884+0000 7f2f2daf0640 1 -- 172.21.15.170:0/3640464952 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f0c005080 con 0x7f2f280b2790 2024-03-29T11:17:06.864 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.884+0000 7f2f257fa640 1 -- 172.21.15.170:0/3640464952 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f2f10084ca0 con 0x7f2f280b2790 2024-03-29T11:17:06.864 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.885+0000 7f2f257fa640 10 client.4439.objecter ms_dispatch 0x7f2f28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:06.864 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.885+0000 7f2f257fa640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:06.864 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.885+0000 7f2f257fa640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:06.864 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.885+0000 7f2f257fa640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-29T11:17:06.864 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.888+0000 7f2f27fff640 10 monclient: get_auth_request con 0x7f2f0806db30 auth_method 0 2024-03-29T11:17:06.865 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.889+0000 7f2f257fa640 10 client.4439.objecter ms_handle_connect 0x7f2f0806db30 2024-03-29T11:17:06.874 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.897+0000 7f2f257fa640 1 -- 172.21.15.170:0/3640464952 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f10050580 con 0x7f2f280b2790 2024-03-29T11:17:06.874 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.897+0000 7f2f257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:06.874 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:06.897+0000 7f2f257fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:07.196 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.219+0000 7f2f2daf0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:07.196 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.219+0000 7f2f2daf0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:07.196 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.219+0000 7f2f2daf0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:07.197 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.219+0000 7f2f2daf0640 1 -- 172.21.15.170:0/3640464952 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f0c005330 con 0x7f2f280b2790 2024-03-29T11:17:07.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.222+0000 7f2f257fa640 1 -- 172.21.15.170:0/3640464952 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f2f100524b0 con 0x7f2f280b2790 2024-03-29T11:17:07.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.222+0000 7f2f257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:07.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.222+0000 7f2f257fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:07.201 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:17:07.201 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:07.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.226+0000 7f2f2daf0640 10 librados: watch_flush enter 2024-03-29T11:17:07.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.226+0000 7f2f2daf0640 10 librados: watch_flush exit 2024-03-29T11:17:07.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.226+0000 7f2f2daf0640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-29T11:17:07.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.226+0000 7f2f2daf0640 10 client.4439.objecter successfully canceled tick 2024-03-29T11:17:07.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.226+0000 7f2f2daf0640 1 -- 172.21.15.170:0/3640464952 >> v1:172.21.15.170:6816/3298994444 conn(0x7f2f0806db30 legacy=0x7f2f0806fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:07.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.226+0000 7f2f2daf0640 10 monclient: shutdown 2024-03-29T11:17:07.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.226+0000 7f2f2daf0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:07.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.226+0000 7f2f2daf0640 1 -- 172.21.15.170:0/3640464952 >> v1:172.21.15.170:6790/0 conn(0x7f2f280b2790 legacy=0x7f2f281daae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:07.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.226+0000 7f2f2daf0640 1 -- 172.21.15.170:0/3640464952 shutdown_connections 2024-03-29T11:17:07.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.226+0000 7f2f2daf0640 1 -- 172.21.15.170:0/3640464952 >> 172.21.15.170:0/3640464952 conn(0x7f2f2806d070 msgr2=0x7f2f28069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:07.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.227+0000 7f2f2daf0640 1 -- 172.21.15.170:0/3640464952 shutdown_connections 2024-03-29T11:17:07.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.227+0000 7f2f2daf0640 1 -- 172.21.15.170:0/3640464952 wait complete. 2024-03-29T11:17:07.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:07.227+0000 7f2f2daf0640 1 librados: shutdown 2024-03-29T11:17:07.222 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:14.029 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:14.184 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.203+0000 7f78e18e9640 10 monclient: get_monmap_and_config 2024-03-29T11:17:14.184 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.203+0000 7f78e18e9640 10 monclient: build_initial_monmap 2024-03-29T11:17:14.184 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.203+0000 7f78e18e9640 10 monclient: monmap: 2024-03-29T11:17:14.184 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:17:14.184 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:14.184 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:17:14.205060+0000 2024-03-29T11:17:14.184 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:17:14.205060+0000 2024-03-29T11:17:14.184 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:14.185 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:14.185 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:17:14.185 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:17:14.185 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:17:14.185 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:14.185 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.205+0000 7f78e18e9640 1 Processor -- start 2024-03-29T11:17:14.185 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.205+0000 7f78e18e9640 1 -- start start 2024-03-29T11:17:14.185 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.205+0000 7f78e18e9640 10 monclient: init 2024-03-29T11:17:14.185 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.206+0000 7f78e18e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:14.188 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.206+0000 7f78e18e9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:17:14.188 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.206+0000 7f78e18e9640 10 monclient(hunting): picked mon.noname-c con 0x7f78d4006800 addr v1:172.21.15.184:6789/0 2024-03-29T11:17:14.188 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.206+0000 7f78e18e9640 10 monclient(hunting): picked mon.noname-b con 0x7f78d4006be0 addr v1:172.21.15.170:6790/0 2024-03-29T11:17:14.189 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.211+0000 7f78e08e7640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f78d4006800 0x7f78d40cad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:46463/0 (socket says 172.21.15.170:46463) 2024-03-29T11:17:14.189 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.212+0000 7f78e18e9640 10 monclient(hunting): picked mon.noname-a con 0x7f78d40324a0 addr v1:172.21.15.170:6789/0 2024-03-29T11:17:14.189 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.212+0000 7f78e18e9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d4032010 con 0x7f78d40324a0 2024-03-29T11:17:14.190 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.212+0000 7f78e18e9640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d402bc00 con 0x7f78d4006800 2024-03-29T11:17:14.190 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.212+0000 7f78e18e9640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d4032960 con 0x7f78d4006be0 2024-03-29T11:17:14.190 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.212+0000 7f78e18e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:14.190 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.212+0000 7f78e18e9640 10 monclient(hunting): authenticate will time out at 1215.524100s 2024-03-29T11:17:14.191 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.212+0000 7f78e08e7640 1 -- 172.21.15.170:0/4094994620 learned_addr learned my addr 172.21.15.170:0/4094994620 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:17:14.193 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.213+0000 7f78db7fe640 1 -- 172.21.15.170:0/4094994620 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916919351 0 0) 0x7f78d402bc00 con 0x7f78d4006800 2024-03-29T11:17:14.193 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.213+0000 7f78db7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:14.193 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.213+0000 7f78db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:14.193 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.213+0000 7f78db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:14.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.213+0000 7f78db7fe640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:17:14.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.213+0000 7f78db7fe640 1 -- 172.21.15.170:0/4094994620 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78cc003520 con 0x7f78d4006800 2024-03-29T11:17:14.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.214+0000 7f78db7fe640 1 -- 172.21.15.170:0/4094994620 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3417126182 0 0) 0x7f78cc003520 con 0x7f78d4006800 2024-03-29T11:17:14.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.214+0000 7f78db7fe640 1 -- 172.21.15.170:0/4094994620 >> v1:172.21.15.170:6790/0 conn(0x7f78d4006be0 legacy=0x7f78d40c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:14.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.214+0000 7f78db7fe640 1 -- 172.21.15.170:0/4094994620 >> v1:172.21.15.170:6789/0 conn(0x7f78d40324a0 legacy=0x7f78d40d30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:14.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.214+0000 7f78db7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:17:14.195 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.214+0000 7f78db7fe640 1 monclient: found mon.noname-c 2024-03-29T11:17:14.195 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.214+0000 7f78db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:14.195 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.214+0000 7f78db7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:17:14.195 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.214+0000 7f78db7fe640 1 -- 172.21.15.170:0/4094994620 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d40cb430 con 0x7f78d4006800 2024-03-29T11:17:14.195 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.214+0000 7f78db7fe640 10 monclient: _finish_auth 0 2024-03-29T11:17:14.195 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.214+0000 7f78db7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:17:14.195 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.214+0000 7f78db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:14.196 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.214+0000 7f78e18e9640 5 monclient: authenticate success, global_id 4411 2024-03-29T11:17:14.196 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.214+0000 7f78e18e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:14.196 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.214+0000 7f78db7fe640 1 -- 172.21.15.170:0/4094994620 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f78dc04f730 con 0x7f78d4006800 2024-03-29T11:17:14.196 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.214+0000 7f78db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:14.196 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.214+0000 7f78db7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:14.196 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.214+0000 7f78db7fe640 10 monclient: dump: 2024-03-29T11:17:14.196 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:14.196 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:14.196 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:14.197 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:14.197 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:14.197 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:14.197 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:14.197 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:14.197 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:14.198 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:14.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.215+0000 7f78e18e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:14.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.215+0000 7f78db7fe640 1 -- 172.21.15.170:0/4094994620 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78dc0008d0 con 0x7f78d4006800 2024-03-29T11:17:14.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.215+0000 7f78db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:14.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.215+0000 7f78db7fe640 1 -- 172.21.15.170:0/4094994620 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f78dc04f730 con 0x7f78d4006800 2024-03-29T11:17:14.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.215+0000 7f78db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:14.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.215+0000 7f78db7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:14.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.215+0000 7f78db7fe640 10 monclient: dump: 2024-03-29T11:17:14.199 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:14.199 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:14.199 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:14.199 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:14.199 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:14.200 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:14.200 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:14.200 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:14.200 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:14.200 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:14.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.215+0000 7f78e18e9640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:14.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.215+0000 7f78e18e9640 4 set_mon_vals no callback set 2024-03-29T11:17:14.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.216+0000 7f78e18e9640 10 monclient: shutdown 2024-03-29T11:17:14.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.216+0000 7f78e18e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:14.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.216+0000 7f78e18e9640 1 -- 172.21.15.170:0/4094994620 >> v1:172.21.15.184:6789/0 conn(0x7f78d4006800 legacy=0x7f78d40cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:14.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.218+0000 7f78e18e9640 1 -- 172.21.15.170:0/4094994620 shutdown_connections 2024-03-29T11:17:14.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.218+0000 7f78e18e9640 1 -- 172.21.15.170:0/4094994620 >> 172.21.15.170:0/4094994620 conn(0x7f78d40058f0 msgr2=0x7f78d4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:14.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.219+0000 7f78e18e9640 1 -- 172.21.15.170:0/4094994620 shutdown_connections 2024-03-29T11:17:14.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.220+0000 7f78e18e9640 1 -- 172.21.15.170:0/4094994620 wait complete. 2024-03-29T11:17:14.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.220+0000 7f78e18e9640 10 monclient: build_initial_monmap 2024-03-29T11:17:14.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.220+0000 7f78e18e9640 10 monclient: monmap: 2024-03-29T11:17:14.205 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:17:14.206 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:14.206 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:17:14.206 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:17:14.206 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:14.206 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:14.206 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:17:14.207 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:17:14.207 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:17:14.207 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:14.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.221+0000 7f78e18e9640 1 librados: starting msgr at 2024-03-29T11:17:14.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.221+0000 7f78e18e9640 1 librados: starting objecter 2024-03-29T11:17:14.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.221+0000 7f78e18e9640 1 Processor -- start 2024-03-29T11:17:14.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.221+0000 7f78e18e9640 1 -- start start 2024-03-29T11:17:14.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.221+0000 7f78e18e9640 1 librados: setting wanted keys 2024-03-29T11:17:14.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.221+0000 7f78e18e9640 1 librados: calling monclient init 2024-03-29T11:17:14.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.221+0000 7f78e18e9640 10 monclient: init 2024-03-29T11:17:14.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.222+0000 7f78e18e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:14.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.222+0000 7f78e18e9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:17:14.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.222+0000 7f78e18e9640 10 monclient(hunting): picked mon.noname-c con 0x7f78d4006800 addr v1:172.21.15.170:6790/0 2024-03-29T11:17:14.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.222+0000 7f78e18e9640 10 monclient(hunting): picked mon.noname-a con 0x7f78d40324a0 addr v1:172.21.15.170:6789/0 2024-03-29T11:17:14.218 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.224+0000 7f78e18e9640 10 monclient(hunting): picked mon.noname-b con 0x7f78d40c55a0 addr v1:172.21.15.184:6789/0 2024-03-29T11:17:14.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.224+0000 7f78e18e9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d40c7ec0 con 0x7f78d40324a0 2024-03-29T11:17:14.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.224+0000 7f78e18e9640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d40c5a60 con 0x7f78d40c55a0 2024-03-29T11:17:14.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.224+0000 7f78e18e9640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d40c8300 con 0x7f78d4006800 2024-03-29T11:17:14.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.224+0000 7f78e18e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:14.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.224+0000 7f78e18e9640 10 monclient(hunting): authenticate will time out at 1215.536481s 2024-03-29T11:17:14.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.225+0000 7f78e08e7640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f78d4006800 0x7f78d40c3490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:59983/0 (socket says 172.21.15.170:59983) 2024-03-29T11:17:14.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.225+0000 7f78e08e7640 1 -- 172.21.15.170:0/2024626982 learned_addr learned my addr 172.21.15.170:0/2024626982 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:17:14.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.227+0000 7f78d97fa640 10 client.?.objecter ms_handle_connect 0x7f78d4006800 2024-03-29T11:17:14.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.227+0000 7f78d97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:14.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.227+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553297072 0 0) 0x7f78d40c8300 con 0x7f78d4006800 2024-03-29T11:17:14.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.227+0000 7f78d97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:17:14.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.227+0000 7f78d97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:14.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.227+0000 7f78d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:14.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.227+0000 7f78d97fa640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:17:14.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.227+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78c0004eb0 con 0x7f78d4006800 2024-03-29T11:17:14.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.228+0000 7f78d97fa640 10 client.?.objecter ms_handle_connect 0x7f78d40c55a0 2024-03-29T11:17:14.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.228+0000 7f78d97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:14.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.228+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253189976 0 0) 0x7f78d40c5a60 con 0x7f78d40c55a0 2024-03-29T11:17:14.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.228+0000 7f78d97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:17:14.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.228+0000 7f78d97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:14.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.228+0000 7f78d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:14.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.228+0000 7f78d97fa640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:17:14.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.228+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78d40c8300 con 0x7f78d40c55a0 2024-03-29T11:17:14.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.230+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 110813221 0 0) 0x7f78c0004eb0 con 0x7f78d4006800 2024-03-29T11:17:14.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.230+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78d40c5a60 con 0x7f78d4006800 2024-03-29T11:17:14.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3503990809 0 0) 0x7f78d40c8300 con 0x7f78d40c55a0 2024-03-29T11:17:14.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78c0004eb0 con 0x7f78d40c55a0 2024-03-29T11:17:14.221 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f78dc04f730 con 0x7f78d4006800 2024-03-29T11:17:14.221 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:14.221 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:14.221 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 10 monclient(hunting): dump: 2024-03-29T11:17:14.221 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:14.221 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:14.221 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:14.221 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:14.221 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:14.221 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:14.221 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:14.221 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:14.221 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:14.221 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:14.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:14.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f78c80017c0 con 0x7f78d40c55a0 2024-03-29T11:17:14.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:14.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:14.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 10 monclient(hunting): dump: 2024-03-29T11:17:14.222 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:14.222 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:14.222 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:14.222 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:14.222 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:14.222 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:14.222 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:14.222 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:14.222 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:14.223 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:14.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 10 client.?.objecter ms_handle_connect 0x7f78d40324a0 2024-03-29T11:17:14.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:14.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3492858183 0 0) 0x7f78c0004eb0 con 0x7f78d40c55a0 2024-03-29T11:17:14.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 >> v1:172.21.15.170:6790/0 conn(0x7f78d4006800 legacy=0x7f78d40c3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:14.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 >> v1:172.21.15.170:6789/0 conn(0x7f78d40324a0 legacy=0x7f78d40c6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:14.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:17:14.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 1 monclient: found mon.b 2024-03-29T11:17:14.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:14.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:17:14.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d40bd180 con 0x7f78d40c55a0 2024-03-29T11:17:14.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 10 monclient: _finish_auth 0 2024-03-29T11:17:14.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:17:14.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:14.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78e18e9640 5 monclient: authenticate success, global_id 4414 2024-03-29T11:17:14.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78e18e9640 10 monclient: _renew_subs 2024-03-29T11:17:14.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78e18e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:17:14.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78e18e9640 1 -- 172.21.15.170:0/2024626982 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d402fd90 con 0x7f78d40c55a0 2024-03-29T11:17:14.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78e18e9640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:14.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78e18e9640 10 monclient: _renew_subs 2024-03-29T11:17:14.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78e18e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:17:14.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78e18e9640 1 -- 172.21.15.170:0/2024626982 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d416d140 con 0x7f78d40c55a0 2024-03-29T11:17:14.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.231+0000 7f78e18e9640 1 librados: init done 2024-03-29T11:17:14.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.236+0000 7f78e18e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:14.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.236+0000 7f78e18e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:14.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.236+0000 7f78e18e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:17:14.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.236+0000 7f78e18e9640 1 -- 172.21.15.170:0/2024626982 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d40c8490 con 0x7f78d40c55a0 2024-03-29T11:17:14.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.236+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78c8001cd0 con 0x7f78d40c55a0 2024-03-29T11:17:14.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.236+0000 7f78d97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:14.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.236+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f78c8003790 con 0x7f78d40c55a0 2024-03-29T11:17:14.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.236+0000 7f78d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:14.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.236+0000 7f78d97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:14.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.236+0000 7f78d97fa640 10 monclient: dump: 2024-03-29T11:17:14.225 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:14.225 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:14.225 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:14.225 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:14.225 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:14.225 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:14.225 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:14.225 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:14.225 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:14.225 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:14.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.236+0000 7f78daffd640 4 set_mon_vals no callback set 2024-03-29T11:17:14.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.237+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f78c80039b0 con 0x7f78d40c55a0 2024-03-29T11:17:14.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.237+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f78c80836a0 con 0x7f78d40c55a0 2024-03-29T11:17:14.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.237+0000 7f78d97fa640 10 client.4414.objecter ms_dispatch 0x7f78dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:14.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.238+0000 7f78d97fa640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:14.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.238+0000 7f78d97fa640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:14.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.238+0000 7f78d97fa640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-29T11:17:14.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.238+0000 7f78e08e7640 10 monclient: get_auth_request con 0x7f78c006dd90 auth_method 0 2024-03-29T11:17:14.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.238+0000 7f78d97fa640 10 client.4414.objecter ms_handle_connect 0x7f78c006dd90 2024-03-29T11:17:14.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.246+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f78c804ef80 con 0x7f78d40c55a0 2024-03-29T11:17:14.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.246+0000 7f78d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:14.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.246+0000 7f78d97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:14.549 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.572+0000 7f78e18e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:14.550 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.572+0000 7f78e18e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:14.550 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.572+0000 7f78e18e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:17:14.550 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.572+0000 7f78e18e9640 1 -- 172.21.15.170:0/2024626982 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78d40bd310 con 0x7f78d40c55a0 2024-03-29T11:17:14.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.574+0000 7f78d97fa640 1 -- 172.21.15.170:0/2024626982 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f78c8050eb0 con 0x7f78d40c55a0 2024-03-29T11:17:14.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.574+0000 7f78d97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:14.555 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:17:14.555 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:14.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.574+0000 7f78d97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:14.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.578+0000 7f78e18e9640 10 librados: watch_flush enter 2024-03-29T11:17:14.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.579+0000 7f78e18e9640 10 librados: watch_flush exit 2024-03-29T11:17:14.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.579+0000 7f78e18e9640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-29T11:17:14.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.579+0000 7f78e18e9640 10 client.4414.objecter successfully canceled tick 2024-03-29T11:17:14.560 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.582+0000 7f78e18e9640 1 -- 172.21.15.170:0/2024626982 >> v1:172.21.15.170:6816/3298994444 conn(0x7f78c006dd90 legacy=0x7f78c0070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:14.560 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.582+0000 7f78e18e9640 10 monclient: shutdown 2024-03-29T11:17:14.560 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.582+0000 7f78e18e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:14.560 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.582+0000 7f78e18e9640 1 -- 172.21.15.170:0/2024626982 >> v1:172.21.15.184:6789/0 conn(0x7f78d40c55a0 legacy=0x7f78d40bca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:14.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.583+0000 7f78e18e9640 1 -- 172.21.15.170:0/2024626982 shutdown_connections 2024-03-29T11:17:14.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.583+0000 7f78e18e9640 1 -- 172.21.15.170:0/2024626982 >> 172.21.15.170:0/2024626982 conn(0x7f78d40058f0 msgr2=0x7f78d40c4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:14.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.583+0000 7f78e18e9640 1 -- 172.21.15.170:0/2024626982 shutdown_connections 2024-03-29T11:17:14.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.583+0000 7f78e18e9640 1 -- 172.21.15.170:0/2024626982 wait complete. 2024-03-29T11:17:14.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:14.583+0000 7f78e18e9640 1 librados: shutdown 2024-03-29T11:17:14.582 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:21.389 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:21.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.574+0000 7f144d154640 10 monclient: get_monmap_and_config 2024-03-29T11:17:21.553 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.574+0000 7f144d154640 10 monclient: build_initial_monmap 2024-03-29T11:17:21.553 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.574+0000 7f144d154640 10 monclient: monmap: 2024-03-29T11:17:21.553 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:17:21.553 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:21.553 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:17:21.576304+0000 2024-03-29T11:17:21.553 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:17:21.576304+0000 2024-03-29T11:17:21.553 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:21.553 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:21.553 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:17:21.553 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:17:21.554 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:17:21.554 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:21.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.575+0000 7f144d154640 1 Processor -- start 2024-03-29T11:17:21.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.576+0000 7f144d154640 1 -- start start 2024-03-29T11:17:21.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.576+0000 7f144d154640 10 monclient: init 2024-03-29T11:17:21.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.576+0000 7f144d154640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:21.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.576+0000 7f144d154640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:17:21.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.577+0000 7f144d154640 10 monclient(hunting): picked mon.noname-c con 0x7f14481422b0 addr v1:172.21.15.184:6789/0 2024-03-29T11:17:21.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.577+0000 7f144d154640 10 monclient(hunting): picked mon.noname-b con 0x7f1448142690 addr v1:172.21.15.170:6790/0 2024-03-29T11:17:21.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.577+0000 7f144d154640 10 monclient(hunting): picked mon.noname-a con 0x7f14480b17d0 addr v1:172.21.15.170:6789/0 2024-03-29T11:17:21.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.577+0000 7f144d154640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1448142c80 con 0x7f14480b17d0 2024-03-29T11:17:21.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.577+0000 7f144d154640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14480aae40 con 0x7f14481422b0 2024-03-29T11:17:21.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.577+0000 7f144d154640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f144813fbd0 con 0x7f1448142690 2024-03-29T11:17:21.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.577+0000 7f144d154640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:21.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.577+0000 7f144d154640 10 monclient(hunting): authenticate will time out at 1222.889122s 2024-03-29T11:17:21.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.577+0000 7f144759e640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f14480b17d0 0x7f14480b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:43938/0 (socket says 172.21.15.170:43938) 2024-03-29T11:17:21.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.577+0000 7f144759e640 1 -- 172.21.15.170:0/756932353 learned_addr learned my addr 172.21.15.170:0/756932353 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:17:21.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.578+0000 7f1445d9b640 1 -- 172.21.15.170:0/756932353 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610333996 0 0) 0x7f144813fbd0 con 0x7f1448142690 2024-03-29T11:17:21.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.578+0000 7f1445d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:17:21.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.578+0000 7f1445d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:21.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.578+0000 7f1445d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:21.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.578+0000 7f1445d9b640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:17:21.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.578+0000 7f1445d9b640 1 -- 172.21.15.170:0/756932353 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1430003520 con 0x7f1448142690 2024-03-29T11:17:21.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.578+0000 7f1445d9b640 1 -- 172.21.15.170:0/756932353 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210927116 0 0) 0x7f14480aae40 con 0x7f14481422b0 2024-03-29T11:17:21.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.578+0000 7f1445d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:17:21.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.578+0000 7f1445d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:21.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.578+0000 7f1445d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:21.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.578+0000 7f1445d9b640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:17:21.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.578+0000 7f1445d9b640 1 -- 172.21.15.170:0/756932353 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f144813fbd0 con 0x7f14481422b0 2024-03-29T11:17:21.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.578+0000 7f1445d9b640 1 -- 172.21.15.170:0/756932353 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585435569 0 0) 0x7f1448142c80 con 0x7f14480b17d0 2024-03-29T11:17:21.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.578+0000 7f1445d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:17:21.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.578+0000 7f1445d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:21.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.578+0000 7f1445d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:21.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.578+0000 7f1445d9b640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:17:21.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.578+0000 7f1445d9b640 1 -- 172.21.15.170:0/756932353 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14480aae40 con 0x7f14480b17d0 2024-03-29T11:17:21.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 1 -- 172.21.15.170:0/756932353 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2017502883 0 0) 0x7f14480aae40 con 0x7f14480b17d0 2024-03-29T11:17:21.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 1 -- 172.21.15.170:0/756932353 >> v1:172.21.15.170:6790/0 conn(0x7f1448142690 legacy=0x7f14480b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:21.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 1 -- 172.21.15.170:0/756932353 >> v1:172.21.15.184:6789/0 conn(0x7f14481422b0 legacy=0x7f144813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:21.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:17:21.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 1 monclient: found mon.noname-a 2024-03-29T11:17:21.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:21.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:17:21.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 1 -- 172.21.15.170:0/756932353 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1448151940 con 0x7f14480b17d0 2024-03-29T11:17:21.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 10 monclient: _finish_auth 0 2024-03-29T11:17:21.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:17:21.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:21.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 1 -- 172.21.15.170:0/756932353 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f1440001930 con 0x7f14480b17d0 2024-03-29T11:17:21.560 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:21.560 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:21.560 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 10 monclient: dump: 2024-03-29T11:17:21.560 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:21.560 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:21.560 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:21.560 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:21.560 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:21.560 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:21.560 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:21.560 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:21.560 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:21.560 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:21.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f144d154640 5 monclient: authenticate success, global_id 4422 2024-03-29T11:17:21.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f144d154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:21.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 1 -- 172.21.15.170:0/756932353 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1440001a90 con 0x7f14480b17d0 2024-03-29T11:17:21.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:21.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 1 -- 172.21.15.170:0/756932353 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f1440001ee0 con 0x7f14480b17d0 2024-03-29T11:17:21.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:21.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:21.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f1445d9b640 10 monclient: dump: 2024-03-29T11:17:21.561 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:21.561 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:21.561 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:21.561 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:21.561 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:21.562 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:21.562 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:21.562 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:21.562 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:21.562 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:21.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f144d154640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:21.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.579+0000 7f144d154640 4 set_mon_vals no callback set 2024-03-29T11:17:21.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.580+0000 7f144d154640 10 monclient: shutdown 2024-03-29T11:17:21.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.580+0000 7f144d154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:21.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.580+0000 7f144d154640 1 -- 172.21.15.170:0/756932353 >> v1:172.21.15.170:6789/0 conn(0x7f14480b17d0 legacy=0x7f14480b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:21.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.580+0000 7f144d154640 1 -- 172.21.15.170:0/756932353 shutdown_connections 2024-03-29T11:17:21.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.580+0000 7f144d154640 1 -- 172.21.15.170:0/756932353 >> 172.21.15.170:0/756932353 conn(0x7f1448066a00 msgr2=0x7f1448066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:21.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.580+0000 7f144d154640 1 -- 172.21.15.170:0/756932353 shutdown_connections 2024-03-29T11:17:21.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.581+0000 7f144d154640 1 -- 172.21.15.170:0/756932353 wait complete. 2024-03-29T11:17:21.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.582+0000 7f144d154640 10 monclient: build_initial_monmap 2024-03-29T11:17:21.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.582+0000 7f144d154640 10 monclient: monmap: 2024-03-29T11:17:21.570 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:17:21.570 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:21.570 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:17:21.570 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:17:21.570 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:21.570 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:21.570 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:17:21.570 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:17:21.570 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:17:21.571 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:21.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.582+0000 7f144d154640 1 librados: starting msgr at 2024-03-29T11:17:21.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.582+0000 7f144d154640 1 librados: starting objecter 2024-03-29T11:17:21.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.583+0000 7f144d154640 1 Processor -- start 2024-03-29T11:17:21.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.583+0000 7f144d154640 1 -- start start 2024-03-29T11:17:21.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.583+0000 7f144d154640 1 librados: setting wanted keys 2024-03-29T11:17:21.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.583+0000 7f144d154640 1 librados: calling monclient init 2024-03-29T11:17:21.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.583+0000 7f144d154640 10 monclient: init 2024-03-29T11:17:21.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.584+0000 7f144d154640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:21.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.584+0000 7f144d154640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:21.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.584+0000 7f144d154640 10 monclient(hunting): picked mon.noname-b con 0x7f14480b17d0 addr v1:172.21.15.184:6789/0 2024-03-29T11:17:21.571 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.584+0000 7f144d154640 10 monclient(hunting): picked mon.noname-a con 0x7f14481422b0 addr v1:172.21.15.170:6789/0 2024-03-29T11:17:21.572 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.584+0000 7f144d154640 10 monclient(hunting): picked mon.noname-c con 0x7f1448142690 addr v1:172.21.15.170:6790/0 2024-03-29T11:17:21.572 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.584+0000 7f144d154640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f144812e900 con 0x7f14481422b0 2024-03-29T11:17:21.572 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.584+0000 7f144d154640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f144814e550 con 0x7f14480b17d0 2024-03-29T11:17:21.572 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.584+0000 7f144d154640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14480ab670 con 0x7f1448142690 2024-03-29T11:17:21.572 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.584+0000 7f144d154640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:21.572 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.584+0000 7f144d154640 10 monclient(hunting): authenticate will time out at 1222.896660s 2024-03-29T11:17:21.572 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.584+0000 7f1446d9d640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f14480b17d0 0x7f144812e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:45442/0 (socket says 172.21.15.170:45442) 2024-03-29T11:17:21.572 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.584+0000 7f1446d9d640 1 -- 172.21.15.170:0/3678219248 learned_addr learned my addr 172.21.15.170:0/3678219248 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:17:21.572 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.584+0000 7f142f7fe640 10 client.?.objecter ms_handle_connect 0x7f1448142690 2024-03-29T11:17:21.572 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.584+0000 7f142f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:21.572 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.584+0000 7f142f7fe640 10 client.?.objecter ms_handle_connect 0x7f14480b17d0 2024-03-29T11:17:21.572 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.584+0000 7f142f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:21.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.585+0000 7f142f7fe640 1 -- 172.21.15.170:0/3678219248 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723743510 0 0) 0x7f144814e550 con 0x7f14480b17d0 2024-03-29T11:17:21.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.585+0000 7f142f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:21.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.585+0000 7f142f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:21.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.585+0000 7f142f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:21.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.585+0000 7f142f7fe640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:17:21.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.585+0000 7f142f7fe640 1 -- 172.21.15.170:0/3678219248 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1434004eb0 con 0x7f14480b17d0 2024-03-29T11:17:21.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.585+0000 7f142f7fe640 10 client.?.objecter ms_handle_connect 0x7f14481422b0 2024-03-29T11:17:21.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.585+0000 7f142f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:21.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.585+0000 7f142f7fe640 1 -- 172.21.15.170:0/3678219248 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2098755260 0 0) 0x7f1434004eb0 con 0x7f14480b17d0 2024-03-29T11:17:21.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 1 -- 172.21.15.170:0/3678219248 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f144814e550 con 0x7f14480b17d0 2024-03-29T11:17:21.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 1 -- 172.21.15.170:0/3678219248 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f14380019c0 con 0x7f14480b17d0 2024-03-29T11:17:21.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:21.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:21.574 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 10 monclient(hunting): dump: 2024-03-29T11:17:21.574 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:21.574 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:21.574 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:21.574 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:21.574 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:21.574 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:21.574 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:21.574 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:21.574 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:21.574 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:21.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:21.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 1 -- 172.21.15.170:0/3678219248 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368239633 0 0) 0x7f144812e900 con 0x7f14481422b0 2024-03-29T11:17:21.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:21.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:21.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:21.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:17:21.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 1 -- 172.21.15.170:0/3678219248 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1434004eb0 con 0x7f14481422b0 2024-03-29T11:17:21.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 1 -- 172.21.15.170:0/3678219248 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3513194944 0 0) 0x7f144814e550 con 0x7f14480b17d0 2024-03-29T11:17:21.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 1 -- 172.21.15.170:0/3678219248 >> v1:172.21.15.170:6790/0 conn(0x7f1448142690 legacy=0x7f144814de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:21.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 1 -- 172.21.15.170:0/3678219248 >> v1:172.21.15.170:6789/0 conn(0x7f14481422b0 legacy=0x7f144814d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:21.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:17:21.575 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 1 monclient: found mon.b 2024-03-29T11:17:21.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:21.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:17:21.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 1 -- 172.21.15.170:0/3678219248 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14480acd20 con 0x7f14480b17d0 2024-03-29T11:17:21.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:17:21.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:17:21.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f142f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:21.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f144d154640 5 monclient: authenticate success, global_id 4420 2024-03-29T11:17:21.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f144d154640 10 monclient: _renew_subs 2024-03-29T11:17:21.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f144d154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:17:21.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.586+0000 7f144d154640 1 -- 172.21.15.170:0/3678219248 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14480ad000 con 0x7f14480b17d0 2024-03-29T11:17:21.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.587+0000 7f144d154640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:21.576 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.587+0000 7f144d154640 10 monclient: _renew_subs 2024-03-29T11:17:21.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.587+0000 7f144d154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:17:21.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.587+0000 7f144d154640 1 -- 172.21.15.170:0/3678219248 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1448132060 con 0x7f14480b17d0 2024-03-29T11:17:21.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.587+0000 7f144d154640 1 librados: init done 2024-03-29T11:17:21.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.587+0000 7f142f7fe640 1 -- 172.21.15.170:0/3678219248 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1438001b20 con 0x7f14480b17d0 2024-03-29T11:17:21.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.587+0000 7f142f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:21.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.587+0000 7f142f7fe640 1 -- 172.21.15.170:0/3678219248 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f1438003a60 con 0x7f14480b17d0 2024-03-29T11:17:21.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.587+0000 7f142f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:21.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.587+0000 7f142f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:21.577 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.587+0000 7f142f7fe640 10 monclient: dump: 2024-03-29T11:17:21.577 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:21.577 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:21.577 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:21.577 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:21.578 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:21.578 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:21.578 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:21.578 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:21.578 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:21.578 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:21.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.587+0000 7f144559a640 4 set_mon_vals no callback set 2024-03-29T11:17:21.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.590+0000 7f142f7fe640 1 -- 172.21.15.170:0/3678219248 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f1438003bc0 con 0x7f14480b17d0 2024-03-29T11:17:21.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.590+0000 7f144d154640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:21.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.590+0000 7f144d154640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:21.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.590+0000 7f144d154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:17:21.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.590+0000 7f144d154640 1 -- 172.21.15.170:0/3678219248 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1428005080 con 0x7f14480b17d0 2024-03-29T11:17:21.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.591+0000 7f142f7fe640 1 -- 172.21.15.170:0/3678219248 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f1438084af0 con 0x7f14480b17d0 2024-03-29T11:17:21.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.592+0000 7f144659c640 10 monclient: get_auth_request con 0x7f143406dd70 auth_method 0 2024-03-29T11:17:21.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.592+0000 7f142f7fe640 10 client.4420.objecter ms_dispatch 0x7f1448000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:21.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.592+0000 7f142f7fe640 3 client.4420.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:21.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.592+0000 7f142f7fe640 3 client.4420.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:21.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.592+0000 7f142f7fe640 20 client.4420.objecter dump_active .. 0 homeless 2024-03-29T11:17:21.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.592+0000 7f142f7fe640 10 client.4420.objecter ms_handle_connect 0x7f143406dd70 2024-03-29T11:17:21.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.598+0000 7f142f7fe640 1 -- 172.21.15.170:0/3678219248 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f14380503d0 con 0x7f14480b17d0 2024-03-29T11:17:21.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.598+0000 7f142f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:21.581 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:21.598+0000 7f142f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:22.013 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.035+0000 7f144d154640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:22.013 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.035+0000 7f144d154640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:22.013 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.035+0000 7f144d154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:17:22.013 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.035+0000 7f144d154640 1 -- 172.21.15.170:0/3678219248 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1428005330 con 0x7f14480b17d0 2024-03-29T11:17:22.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.036+0000 7f142f7fe640 1 -- 172.21.15.170:0/3678219248 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f1438052300 con 0x7f14480b17d0 2024-03-29T11:17:22.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.036+0000 7f142f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:22.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.036+0000 7f142f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:22.015 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:17:22.015 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:22.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.040+0000 7f144d154640 10 librados: watch_flush enter 2024-03-29T11:17:22.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.041+0000 7f144d154640 10 librados: watch_flush exit 2024-03-29T11:17:22.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.041+0000 7f144d154640 20 client.4420.objecter shutdown clearing up homeless session... 2024-03-29T11:17:22.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.041+0000 7f144d154640 10 client.4420.objecter successfully canceled tick 2024-03-29T11:17:22.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.041+0000 7f144d154640 1 -- 172.21.15.170:0/3678219248 >> v1:172.21.15.170:6816/3298994444 conn(0x7f143406dd70 legacy=0x7f1434070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:22.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.041+0000 7f144d154640 10 monclient: shutdown 2024-03-29T11:17:22.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.041+0000 7f144d154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:22.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.041+0000 7f144d154640 1 -- 172.21.15.170:0/3678219248 >> v1:172.21.15.184:6789/0 conn(0x7f14480b17d0 legacy=0x7f144812e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:22.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.041+0000 7f144d154640 1 -- 172.21.15.170:0/3678219248 shutdown_connections 2024-03-29T11:17:22.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.041+0000 7f144d154640 1 -- 172.21.15.170:0/3678219248 >> 172.21.15.170:0/3678219248 conn(0x7f1448066a00 msgr2=0x7f144813d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:22.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.041+0000 7f144d154640 1 -- 172.21.15.170:0/3678219248 shutdown_connections 2024-03-29T11:17:22.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.041+0000 7f144d154640 1 -- 172.21.15.170:0/3678219248 wait complete. 2024-03-29T11:17:22.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:22.042+0000 7f144d154640 1 librados: shutdown 2024-03-29T11:17:22.035 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:28.842 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:29.005 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.027+0000 7f1956c63640 10 monclient: get_monmap_and_config 2024-03-29T11:17:29.005 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.027+0000 7f1956c63640 10 monclient: build_initial_monmap 2024-03-29T11:17:29.005 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.027+0000 7f1956c63640 10 monclient: monmap: 2024-03-29T11:17:29.005 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:17:29.005 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:29.005 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:17:29.028938+0000 2024-03-29T11:17:29.006 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:17:29.028938+0000 2024-03-29T11:17:29.006 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:29.006 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:29.006 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:17:29.006 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:17:29.006 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:17:29.006 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:29.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.029+0000 7f1956c63640 1 Processor -- start 2024-03-29T11:17:29.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.029+0000 7f1956c63640 1 -- start start 2024-03-29T11:17:29.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.029+0000 7f1956c63640 10 monclient: init 2024-03-29T11:17:29.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.030+0000 7f1956c63640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:29.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.030+0000 7f1956c63640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:17:29.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.030+0000 7f1956c63640 10 monclient(hunting): picked mon.noname-c con 0x7f19500b4330 addr v1:172.21.15.184:6789/0 2024-03-29T11:17:29.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.030+0000 7f1956c63640 10 monclient(hunting): picked mon.noname-b con 0x7f19501470b0 addr v1:172.21.15.170:6790/0 2024-03-29T11:17:29.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.030+0000 7f1956c63640 10 monclient(hunting): picked mon.noname-a con 0x7f19501439b0 addr v1:172.21.15.170:6789/0 2024-03-29T11:17:29.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.030+0000 7f1956c63640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195006f2a0 con 0x7f19501439b0 2024-03-29T11:17:29.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.030+0000 7f1956c63640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1950146b90 con 0x7f19500b4330 2024-03-29T11:17:29.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.030+0000 7f1956c63640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19500b2ca0 con 0x7f19501470b0 2024-03-29T11:17:29.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.030+0000 7f1956c63640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:29.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.030+0000 7f1956c63640 10 monclient(hunting): authenticate will time out at 1230.342675s 2024-03-29T11:17:29.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.033+0000 7f194ffff640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f19501470b0 0x7f19500b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:46026/0 (socket says 172.21.15.170:46026) 2024-03-29T11:17:29.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.033+0000 7f194ffff640 1 -- 172.21.15.170:0/2501978559 learned_addr learned my addr 172.21.15.170:0/2501978559 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:17:29.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.037+0000 7f194f7fe640 1 -- 172.21.15.170:0/2501978559 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642469342 0 0) 0x7f19500b2ca0 con 0x7f19501470b0 2024-03-29T11:17:29.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.037+0000 7f194f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:29.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.037+0000 7f194f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:29.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.037+0000 7f194f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:29.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.037+0000 7f194f7fe640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:17:29.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.037+0000 7f194f7fe640 1 -- 172.21.15.170:0/2501978559 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1938003520 con 0x7f19501470b0 2024-03-29T11:17:29.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.037+0000 7f194f7fe640 1 -- 172.21.15.170:0/2501978559 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691064854 0 0) 0x7f1950146b90 con 0x7f19500b4330 2024-03-29T11:17:29.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.037+0000 7f194f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:29.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.037+0000 7f194f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:29.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.037+0000 7f194f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:29.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.037+0000 7f194f7fe640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:17:29.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.037+0000 7f194f7fe640 1 -- 172.21.15.170:0/2501978559 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19500b2ca0 con 0x7f19500b4330 2024-03-29T11:17:29.018 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f194f7fe640 1 -- 172.21.15.170:0/2501978559 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3802081356 0 0) 0x7f19500b2ca0 con 0x7f19500b4330 2024-03-29T11:17:29.018 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f194f7fe640 1 -- 172.21.15.170:0/2501978559 >> v1:172.21.15.170:6790/0 conn(0x7f19501470b0 legacy=0x7f19500b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:29.019 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f194f7fe640 1 -- 172.21.15.170:0/2501978559 >> v1:172.21.15.170:6789/0 conn(0x7f19501439b0 legacy=0x7f1950151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:29.019 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f194f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:17:29.019 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f194f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:17:29.019 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f194f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:29.019 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f194f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:17:29.019 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f194f7fe640 1 -- 172.21.15.170:0/2501978559 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195006e0c0 con 0x7f19500b4330 2024-03-29T11:17:29.019 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f194f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:17:29.019 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f194f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:17:29.019 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f194f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:29.019 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f1956c63640 5 monclient: authenticate success, global_id 4423 2024-03-29T11:17:29.019 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f1956c63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:29.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f194f7fe640 1 -- 172.21.15.170:0/2501978559 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f1940001670 con 0x7f19500b4330 2024-03-29T11:17:29.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f194f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:29.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f194f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:29.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f194f7fe640 10 monclient: dump: 2024-03-29T11:17:29.020 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:29.020 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:29.020 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:29.020 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:29.020 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:29.020 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:29.020 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:29.020 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:29.020 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:29.021 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:29.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f1956c63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:29.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f194f7fe640 1 -- 172.21.15.170:0/2501978559 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1940001890 con 0x7f19500b4330 2024-03-29T11:17:29.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f194f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:29.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f1956c63640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:29.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.041+0000 7f1956c63640 4 set_mon_vals no callback set 2024-03-29T11:17:29.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.042+0000 7f1956c63640 10 monclient: shutdown 2024-03-29T11:17:29.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.042+0000 7f1956c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:29.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.042+0000 7f1956c63640 1 -- 172.21.15.170:0/2501978559 >> v1:172.21.15.184:6789/0 conn(0x7f19500b4330 legacy=0x7f19500b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:29.023 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.045+0000 7f1956c63640 1 -- 172.21.15.170:0/2501978559 shutdown_connections 2024-03-29T11:17:29.023 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.045+0000 7f1956c63640 1 -- 172.21.15.170:0/2501978559 >> 172.21.15.170:0/2501978559 conn(0x7f1950066a00 msgr2=0x7f1950066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:29.023 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.045+0000 7f1956c63640 1 -- 172.21.15.170:0/2501978559 shutdown_connections 2024-03-29T11:17:29.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.046+0000 7f1956c63640 1 -- 172.21.15.170:0/2501978559 wait complete. 2024-03-29T11:17:29.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.047+0000 7f1956c63640 10 monclient: build_initial_monmap 2024-03-29T11:17:29.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.047+0000 7f1956c63640 10 monclient: monmap: 2024-03-29T11:17:29.026 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:17:29.026 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:29.026 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:17:29.026 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:17:29.026 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:29.026 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:29.026 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:17:29.027 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:17:29.027 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:17:29.027 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:29.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.047+0000 7f1956c63640 1 librados: starting msgr at 2024-03-29T11:17:29.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.047+0000 7f1956c63640 1 librados: starting objecter 2024-03-29T11:17:29.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.048+0000 7f1956c63640 1 Processor -- start 2024-03-29T11:17:29.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.048+0000 7f1956c63640 1 -- start start 2024-03-29T11:17:29.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.048+0000 7f1956c63640 1 librados: setting wanted keys 2024-03-29T11:17:29.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.048+0000 7f1956c63640 1 librados: calling monclient init 2024-03-29T11:17:29.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.048+0000 7f1956c63640 10 monclient: init 2024-03-29T11:17:29.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.049+0000 7f1956c63640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:29.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.049+0000 7f1956c63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:29.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.049+0000 7f1956c63640 10 monclient(hunting): picked mon.noname-b con 0x7f19500b4330 addr v1:172.21.15.184:6789/0 2024-03-29T11:17:29.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.049+0000 7f1956c63640 10 monclient(hunting): picked mon.noname-a con 0x7f19501439b0 addr v1:172.21.15.170:6789/0 2024-03-29T11:17:29.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.049+0000 7f1956c63640 10 monclient(hunting): picked mon.noname-c con 0x7f19501470b0 addr v1:172.21.15.170:6790/0 2024-03-29T11:17:29.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.049+0000 7f1956c63640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19501200f0 con 0x7f19501439b0 2024-03-29T11:17:29.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.049+0000 7f1956c63640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19501d1220 con 0x7f19500b4330 2024-03-29T11:17:29.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.049+0000 7f1956c63640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19501d13b0 con 0x7f19501470b0 2024-03-29T11:17:29.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.049+0000 7f1956c63640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:29.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.049+0000 7f1956c63640 10 monclient(hunting): authenticate will time out at 1230.361792s 2024-03-29T11:17:29.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.050+0000 7f1955200640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f19501470b0 0x7f1950140290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:53818/0 (socket says 172.21.15.170:53818) 2024-03-29T11:17:29.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.050+0000 7f1955200640 1 -- 172.21.15.170:0/4031633332 learned_addr learned my addr 172.21.15.170:0/4031633332 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:17:29.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.051+0000 7f194d7fa640 10 client.?.objecter ms_handle_connect 0x7f19501470b0 2024-03-29T11:17:29.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.051+0000 7f194d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:29.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 10 client.?.objecter ms_handle_connect 0x7f19501439b0 2024-03-29T11:17:29.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:29.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 10 client.?.objecter ms_handle_connect 0x7f19500b4330 2024-03-29T11:17:29.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:29.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820862021 0 0) 0x7f195006f2a0 con 0x7f19501470b0 2024-03-29T11:17:29.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:17:29.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:29.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:29.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:17:29.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f193c004eb0 con 0x7f19501470b0 2024-03-29T11:17:29.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032804224 0 0) 0x7f19501d1220 con 0x7f19500b4330 2024-03-29T11:17:29.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:17:29.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:29.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:29.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:17:29.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f195006f2a0 con 0x7f19500b4330 2024-03-29T11:17:29.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2686547832 0 0) 0x7f19501200f0 con 0x7f19501439b0 2024-03-29T11:17:29.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:17:29.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:29.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:29.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:17:29.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19501d1220 con 0x7f19501439b0 2024-03-29T11:17:29.036 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.055+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2101745341 0 0) 0x7f195006f2a0 con 0x7f19500b4330 2024-03-29T11:17:29.036 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19501200f0 con 0x7f19500b4330 2024-03-29T11:17:29.036 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f1940003790 con 0x7f19500b4330 2024-03-29T11:17:29.036 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:29.036 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:29.036 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 10 monclient(hunting): dump: 2024-03-29T11:17:29.036 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:29.036 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:29.036 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:29.036 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:29.036 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:29.036 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:29.036 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:29.036 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:29.036 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:29.037 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:29.037 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:29.037 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1402123083 0 0) 0x7f193c004eb0 con 0x7f19501470b0 2024-03-29T11:17:29.037 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f195006f2a0 con 0x7f19501470b0 2024-03-29T11:17:29.037 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f1944001770 con 0x7f19501470b0 2024-03-29T11:17:29.037 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:29.037 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:29.037 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 10 monclient(hunting): dump: 2024-03-29T11:17:29.037 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:29.037 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:29.037 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:29.037 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:29.037 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:29.037 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:29.037 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:29.038 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:29.038 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:29.038 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:29.038 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2740618243 0 0) 0x7f19501200f0 con 0x7f19500b4330 2024-03-29T11:17:29.038 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 >> v1:172.21.15.170:6790/0 conn(0x7f19501470b0 legacy=0x7f1950140290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:29.038 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 >> v1:172.21.15.170:6789/0 conn(0x7f19501439b0 legacy=0x7f19501db3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:29.038 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:17:29.038 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 1 monclient: found mon.b 2024-03-29T11:17:29.038 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:29.038 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:17:29.038 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19501dbae0 con 0x7f19500b4330 2024-03-29T11:17:29.038 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:17:29.038 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:17:29.038 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:29.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f1956c63640 5 monclient: authenticate success, global_id 4426 2024-03-29T11:17:29.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f1956c63640 10 monclient: _renew_subs 2024-03-29T11:17:29.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f1956c63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:17:29.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f1956c63640 1 -- 172.21.15.170:0/4031633332 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19501dbd30 con 0x7f19500b4330 2024-03-29T11:17:29.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f1956c63640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:29.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f1956c63640 10 monclient: _renew_subs 2024-03-29T11:17:29.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f1956c63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:17:29.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f1956c63640 1 -- 172.21.15.170:0/4031633332 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19501416b0 con 0x7f19500b4330 2024-03-29T11:17:29.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f1956c63640 1 librados: init done 2024-03-29T11:17:29.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1940001210 con 0x7f19500b4330 2024-03-29T11:17:29.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:29.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.056+0000 7f194effd640 4 set_mon_vals no callback set 2024-03-29T11:17:29.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.057+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f1940001c20 con 0x7f19500b4330 2024-03-29T11:17:29.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.057+0000 7f194d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:29.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.057+0000 7f194d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:29.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.057+0000 7f194d7fa640 10 monclient: dump: 2024-03-29T11:17:29.040 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:29.040 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:29.040 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:29.041 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:29.041 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:29.041 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:29.041 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:29.041 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:29.041 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:29.041 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:29.041 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.058+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f1940019a10 con 0x7f19500b4330 2024-03-29T11:17:29.041 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.059+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f1940083910 con 0x7f19500b4330 2024-03-29T11:17:29.041 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.059+0000 7f194d7fa640 10 client.4426.objecter ms_dispatch 0x7f1950000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:29.041 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.059+0000 7f194d7fa640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:29.041 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.059+0000 7f194d7fa640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:29.041 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.059+0000 7f194d7fa640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-29T11:17:29.041 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.059+0000 7f194ffff640 10 monclient: get_auth_request con 0x7f193c06dfe0 auth_method 0 2024-03-29T11:17:29.041 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.060+0000 7f1956c63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:29.042 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.060+0000 7f1956c63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:29.042 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.060+0000 7f1956c63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:17:29.042 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.060+0000 7f1956c63640 1 -- 172.21.15.170:0/4031633332 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1930005080 con 0x7f19500b4330 2024-03-29T11:17:29.042 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.060+0000 7f194d7fa640 10 client.4426.objecter ms_handle_connect 0x7f193c06dfe0 2024-03-29T11:17:29.045 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.067+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f194004f020 con 0x7f19500b4330 2024-03-29T11:17:29.045 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.067+0000 7f194d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:29.045 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.067+0000 7f194d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:29.412 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.434+0000 7f1956c63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:29.412 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.434+0000 7f1956c63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:29.412 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.434+0000 7f1956c63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:17:29.412 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.434+0000 7f1956c63640 1 -- 172.21.15.170:0/4031633332 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1930005330 con 0x7f19500b4330 2024-03-29T11:17:29.413 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.435+0000 7f194d7fa640 1 -- 172.21.15.170:0/4031633332 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f1940051120 con 0x7f19500b4330 2024-03-29T11:17:29.413 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.435+0000 7f194d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:29.413 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.435+0000 7f194d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:29.416 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:17:29.416 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:29.421 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.442+0000 7f1956c63640 10 librados: watch_flush enter 2024-03-29T11:17:29.421 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.442+0000 7f1956c63640 10 librados: watch_flush exit 2024-03-29T11:17:29.421 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.442+0000 7f1956c63640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-29T11:17:29.421 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.443+0000 7f1956c63640 10 client.4426.objecter successfully canceled tick 2024-03-29T11:17:29.421 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.443+0000 7f1956c63640 1 -- 172.21.15.170:0/4031633332 >> v1:172.21.15.170:6816/3298994444 conn(0x7f193c06dfe0 legacy=0x7f193c0704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:29.421 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.443+0000 7f1956c63640 10 monclient: shutdown 2024-03-29T11:17:29.421 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.443+0000 7f1956c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:29.421 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.443+0000 7f1956c63640 1 -- 172.21.15.170:0/4031633332 >> v1:172.21.15.184:6789/0 conn(0x7f19500b4330 legacy=0x7f195011f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:29.421 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.443+0000 7f1956c63640 1 -- 172.21.15.170:0/4031633332 shutdown_connections 2024-03-29T11:17:29.422 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.443+0000 7f1956c63640 1 -- 172.21.15.170:0/4031633332 >> 172.21.15.170:0/4031633332 conn(0x7f1950066a00 msgr2=0x7f1950142f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:29.422 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.443+0000 7f1956c63640 1 -- 172.21.15.170:0/4031633332 shutdown_connections 2024-03-29T11:17:29.422 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.443+0000 7f1956c63640 1 -- 172.21.15.170:0/4031633332 wait complete. 2024-03-29T11:17:29.422 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:29.443+0000 7f1956c63640 1 librados: shutdown 2024-03-29T11:17:29.443 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:36.251 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:36.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.424+0000 7fa62580e640 10 monclient: get_monmap_and_config 2024-03-29T11:17:36.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.424+0000 7fa62580e640 10 monclient: build_initial_monmap 2024-03-29T11:17:36.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.424+0000 7fa62580e640 10 monclient: monmap: 2024-03-29T11:17:36.401 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:17:36.401 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:36.402 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:17:36.424883+0000 2024-03-29T11:17:36.402 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:17:36.424883+0000 2024-03-29T11:17:36.402 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:36.402 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:36.402 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:17:36.402 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:17:36.402 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:17:36.402 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:36.404 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.425+0000 7fa62580e640 1 Processor -- start 2024-03-29T11:17:36.404 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.425+0000 7fa62580e640 1 -- start start 2024-03-29T11:17:36.404 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.425+0000 7fa62580e640 10 monclient: init 2024-03-29T11:17:36.404 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.426+0000 7fa62580e640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:36.404 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.426+0000 7fa62580e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:17:36.404 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.426+0000 7fa62580e640 10 monclient(hunting): picked mon.noname-a con 0x7fa6201457f0 addr v1:172.21.15.170:6789/0 2024-03-29T11:17:36.405 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.426+0000 7fa62580e640 10 monclient(hunting): picked mon.noname-b con 0x7fa62013f650 addr v1:172.21.15.170:6790/0 2024-03-29T11:17:36.405 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.426+0000 7fa62580e640 10 monclient(hunting): picked mon.noname-c con 0x7fa6200a9d70 addr v1:172.21.15.184:6789/0 2024-03-29T11:17:36.405 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.426+0000 7fa62580e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6200aaef0 con 0x7fa6201457f0 2024-03-29T11:17:36.405 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.426+0000 7fa62580e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa620145bd0 con 0x7fa6200a9d70 2024-03-29T11:17:36.405 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.426+0000 7fa62580e640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa620145eb0 con 0x7fa62013f650 2024-03-29T11:17:36.405 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.426+0000 7fa62580e640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:36.405 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.426+0000 7fa62580e640 10 monclient(hunting): authenticate will time out at 1237.737911s 2024-03-29T11:17:36.405 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.426+0000 7fa61effd640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fa6201457f0 0x7fa62013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:36570/0 (socket says 172.21.15.170:36570) 2024-03-29T11:17:36.405 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.426+0000 7fa61effd640 1 -- 172.21.15.170:0/456243921 learned_addr learned my addr 172.21.15.170:0/456243921 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:17:36.405 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.427+0000 7fa61dffb640 1 -- 172.21.15.170:0/456243921 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848855776 0 0) 0x7fa6200aaef0 con 0x7fa6201457f0 2024-03-29T11:17:36.405 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.427+0000 7fa61dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:36.406 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.427+0000 7fa61dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:36.406 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.427+0000 7fa61dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:36.406 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.427+0000 7fa61dffb640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:17:36.406 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.427+0000 7fa61dffb640 1 -- 172.21.15.170:0/456243921 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa60c003520 con 0x7fa6201457f0 2024-03-29T11:17:36.406 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.427+0000 7fa61dffb640 1 -- 172.21.15.170:0/456243921 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798610863 0 0) 0x7fa620145eb0 con 0x7fa62013f650 2024-03-29T11:17:36.406 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.427+0000 7fa61dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:36.406 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.427+0000 7fa61dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:36.406 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.427+0000 7fa61dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:36.406 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.427+0000 7fa61dffb640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:17:36.406 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.427+0000 7fa61dffb640 1 -- 172.21.15.170:0/456243921 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6200aaef0 con 0x7fa62013f650 2024-03-29T11:17:36.412 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.428+0000 7fa61dffb640 1 -- 172.21.15.170:0/456243921 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2931843696 0 0) 0x7fa60c003520 con 0x7fa6201457f0 2024-03-29T11:17:36.412 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.428+0000 7fa61dffb640 1 -- 172.21.15.170:0/456243921 >> v1:172.21.15.170:6790/0 conn(0x7fa62013f650 legacy=0x7fa6200b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:36.412 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.428+0000 7fa61dffb640 1 -- 172.21.15.170:0/456243921 >> v1:172.21.15.184:6789/0 conn(0x7fa6200a9d70 legacy=0x7fa62014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:17:36.412 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.428+0000 7fa61dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:17:36.412 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.428+0000 7fa61dffb640 1 monclient: found mon.noname-a 2024-03-29T11:17:36.412 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.428+0000 7fa61dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:36.412 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.428+0000 7fa61dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:17:36.412 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.428+0000 7fa61dffb640 1 -- 172.21.15.170:0/456243921 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa620146340 con 0x7fa6201457f0 2024-03-29T11:17:36.412 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.428+0000 7fa61dffb640 10 monclient: _finish_auth 0 2024-03-29T11:17:36.413 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.428+0000 7fa61dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:17:36.413 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.428+0000 7fa61dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:36.413 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.428+0000 7fa62580e640 5 monclient: authenticate success, global_id 4434 2024-03-29T11:17:36.413 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.428+0000 7fa62580e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:36.413 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.428+0000 7fa61dffb640 1 -- 172.21.15.170:0/456243921 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa610001a10 con 0x7fa6201457f0 2024-03-29T11:17:36.413 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.428+0000 7fa61dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:36.413 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.428+0000 7fa61dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:36.413 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.428+0000 7fa61dffb640 10 monclient: dump: 2024-03-29T11:17:36.413 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:36.413 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:36.413 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:36.413 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:36.414 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:36.414 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:36.414 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:36.414 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:36.414 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:36.414 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:36.414 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.428+0000 7fa62580e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:36.414 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.429+0000 7fa61dffb640 1 -- 172.21.15.170:0/456243921 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa610001c30 con 0x7fa6201457f0 2024-03-29T11:17:36.414 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.429+0000 7fa61dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:36.414 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.429+0000 7fa62580e640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:36.415 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.429+0000 7fa62580e640 4 set_mon_vals no callback set 2024-03-29T11:17:36.415 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.429+0000 7fa61dffb640 1 -- 172.21.15.170:0/456243921 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa610001fc0 con 0x7fa6201457f0 2024-03-29T11:17:36.415 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.429+0000 7fa61dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:36.415 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.429+0000 7fa61dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:36.415 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.429+0000 7fa61dffb640 10 monclient: dump: 2024-03-29T11:17:36.415 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:36.415 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:36.415 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:36.415 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:36.415 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:36.415 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:36.415 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:36.416 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:36.416 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:36.416 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:36.416 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.430+0000 7fa62580e640 10 monclient: shutdown 2024-03-29T11:17:36.416 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.430+0000 7fa62580e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:36.416 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.430+0000 7fa62580e640 1 -- 172.21.15.170:0/456243921 >> v1:172.21.15.170:6789/0 conn(0x7fa6201457f0 legacy=0x7fa62013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:36.416 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.430+0000 7fa62580e640 1 -- 172.21.15.170:0/456243921 shutdown_connections 2024-03-29T11:17:36.416 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.430+0000 7fa62580e640 1 -- 172.21.15.170:0/456243921 >> 172.21.15.170:0/456243921 conn(0x7fa620066a00 msgr2=0x7fa620066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:36.416 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.430+0000 7fa62580e640 1 -- 172.21.15.170:0/456243921 shutdown_connections 2024-03-29T11:17:36.416 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.430+0000 7fa62580e640 1 -- 172.21.15.170:0/456243921 wait complete. 2024-03-29T11:17:36.416 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.431+0000 7fa62580e640 10 monclient: build_initial_monmap 2024-03-29T11:17:36.416 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.431+0000 7fa62580e640 10 monclient: monmap: 2024-03-29T11:17:36.416 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:17:36.416 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:36.417 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:17:36.417 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:17:36.417 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:36.417 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:36.417 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:17:36.417 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:17:36.417 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:17:36.417 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:36.417 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.432+0000 7fa62580e640 1 librados: starting msgr at 2024-03-29T11:17:36.417 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.432+0000 7fa62580e640 1 librados: starting objecter 2024-03-29T11:17:36.417 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.432+0000 7fa62580e640 1 Processor -- start 2024-03-29T11:17:36.417 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.432+0000 7fa62580e640 1 -- start start 2024-03-29T11:17:36.417 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.432+0000 7fa62580e640 1 librados: setting wanted keys 2024-03-29T11:17:36.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.432+0000 7fa62580e640 1 librados: calling monclient init 2024-03-29T11:17:36.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.432+0000 7fa62580e640 10 monclient: init 2024-03-29T11:17:36.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.433+0000 7fa62580e640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:36.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.433+0000 7fa62580e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:17:36.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.433+0000 7fa62580e640 10 monclient(hunting): picked mon.noname-a con 0x7fa6200a9d70 addr v1:172.21.15.170:6789/0 2024-03-29T11:17:36.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.433+0000 7fa62580e640 10 monclient(hunting): picked mon.noname-b con 0x7fa62013f650 addr v1:172.21.15.184:6789/0 2024-03-29T11:17:36.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.433+0000 7fa62580e640 10 monclient(hunting): picked mon.noname-c con 0x7fa6201457f0 addr v1:172.21.15.170:6790/0 2024-03-29T11:17:36.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.433+0000 7fa62580e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6201df860 con 0x7fa6200a9d70 2024-03-29T11:17:36.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.433+0000 7fa62580e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6201df230 con 0x7fa62013f650 2024-03-29T11:17:36.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.433+0000 7fa62580e640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6201df3c0 con 0x7fa6201457f0 2024-03-29T11:17:36.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.433+0000 7fa62580e640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:36.418 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.433+0000 7fa62580e640 10 monclient(hunting): authenticate will time out at 1237.744723s 2024-03-29T11:17:36.419 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.433+0000 7fa61e7fc640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fa62013f650 0x7fa6201374c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:53392/0 (socket says 172.21.15.170:53392) 2024-03-29T11:17:36.419 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.433+0000 7fa61e7fc640 1 -- 172.21.15.170:0/2943506583 learned_addr learned my addr 172.21.15.170:0/2943506583 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:17:36.419 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.433+0000 7fa617fff640 10 client.?.objecter ms_handle_connect 0x7fa62013f650 2024-03-29T11:17:36.419 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.433+0000 7fa617fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:36.419 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.433+0000 7fa617fff640 10 client.?.objecter ms_handle_connect 0x7fa6201457f0 2024-03-29T11:17:36.419 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.433+0000 7fa617fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:36.419 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.433+0000 7fa617fff640 10 client.?.objecter ms_handle_connect 0x7fa6200a9d70 2024-03-29T11:17:36.419 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.433+0000 7fa617fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:36.419 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.434+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452703121 0 0) 0x7fa6201df3c0 con 0x7fa6201457f0 2024-03-29T11:17:36.419 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.434+0000 7fa617fff640 10 monclient(hunting): _negotiate 2024-03-29T11:17:36.419 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.434+0000 7fa617fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:36.419 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.434+0000 7fa617fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:36.420 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.434+0000 7fa617fff640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:17:36.420 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.434+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa600004eb0 con 0x7fa6201457f0 2024-03-29T11:17:36.420 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.434+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435878388 0 0) 0x7fa6201df860 con 0x7fa6200a9d70 2024-03-29T11:17:36.420 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.434+0000 7fa617fff640 10 monclient(hunting): _negotiate 2024-03-29T11:17:36.420 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.434+0000 7fa617fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:36.420 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.434+0000 7fa617fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:36.420 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.434+0000 7fa617fff640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:17:36.420 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.434+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6201df3c0 con 0x7fa6200a9d70 2024-03-29T11:17:36.420 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.434+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4208399363 0 0) 0x7fa6201df3c0 con 0x7fa6200a9d70 2024-03-29T11:17:36.420 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.434+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6000064f0 con 0x7fa6200a9d70 2024-03-29T11:17:36.420 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.434+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa610001da0 con 0x7fa6200a9d70 2024-03-29T11:17:36.420 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.434+0000 7fa617fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:36.421 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.434+0000 7fa617fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:36.421 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.434+0000 7fa617fff640 10 monclient(hunting): dump: 2024-03-29T11:17:36.421 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:36.421 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:36.421 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:36.421 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:36.421 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:36.421 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:36.421 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:36.421 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:36.421 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:36.421 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:36.422 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.434+0000 7fa617fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:36.422 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 455037991 0 0) 0x7fa600004eb0 con 0x7fa6201457f0 2024-03-29T11:17:36.422 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6201df3c0 con 0x7fa6201457f0 2024-03-29T11:17:36.422 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa608001510 con 0x7fa6201457f0 2024-03-29T11:17:36.422 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa617fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:36.422 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa617fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:36.422 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa617fff640 10 monclient(hunting): dump: 2024-03-29T11:17:36.422 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:36.422 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:36.422 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:36.422 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:36.422 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:36.422 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:36.423 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:36.423 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:36.423 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:36.423 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:36.423 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4185490974 0 0) 0x7fa6000064f0 con 0x7fa6200a9d70 2024-03-29T11:17:36.423 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 >> v1:172.21.15.170:6790/0 conn(0x7fa6201457f0 legacy=0x7fa6201db760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:36.423 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 >> v1:172.21.15.184:6789/0 conn(0x7fa62013f650 legacy=0x7fa6201374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:36.423 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa617fff640 10 monclient: _finish_hunting 0 2024-03-29T11:17:36.423 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa617fff640 1 monclient: found mon.a 2024-03-29T11:17:36.423 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa617fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:36.423 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa617fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:17:36.423 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6201df550 con 0x7fa6200a9d70 2024-03-29T11:17:36.423 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa617fff640 10 monclient: _finish_auth 0 2024-03-29T11:17:36.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa617fff640 10 monclient: _check_auth_tickets 2024-03-29T11:17:36.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa617fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:36.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa62580e640 5 monclient: authenticate success, global_id 4437 2024-03-29T11:17:36.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa62580e640 10 monclient: _renew_subs 2024-03-29T11:17:36.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa62580e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:17:36.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa62580e640 1 -- 172.21.15.170:0/2943506583 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa620142760 con 0x7fa6200a9d70 2024-03-29T11:17:36.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa62580e640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:36.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa62580e640 10 monclient: _renew_subs 2024-03-29T11:17:36.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa62580e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:17:36.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa62580e640 1 -- 172.21.15.170:0/2943506583 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa620137bd0 con 0x7fa6200a9d70 2024-03-29T11:17:36.424 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.435+0000 7fa62580e640 1 librados: init done 2024-03-29T11:17:36.426 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.436+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa610002000 con 0x7fa6200a9d70 2024-03-29T11:17:36.426 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.436+0000 7fa617fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:36.426 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.436+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa610004fb0 con 0x7fa6200a9d70 2024-03-29T11:17:36.427 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.436+0000 7fa617fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:36.427 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.436+0000 7fa617fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:36.427 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.436+0000 7fa617fff640 10 monclient: dump: 2024-03-29T11:17:36.427 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:36.427 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:36.427 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:36.427 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:36.427 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:36.427 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:36.427 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:36.428 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:36.428 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:36.428 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:36.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.436+0000 7fa61d7fa640 4 set_mon_vals no callback set 2024-03-29T11:17:36.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.436+0000 7fa62580e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:36.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.437+0000 7fa62580e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:36.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.437+0000 7fa62580e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:17:36.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.437+0000 7fa62580e640 1 -- 172.21.15.170:0/2943506583 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa604005080 con 0x7fa6200a9d70 2024-03-29T11:17:36.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.438+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fa61001af80 con 0x7fa6200a9d70 2024-03-29T11:17:36.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.439+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fa6100849e0 con 0x7fa6200a9d70 2024-03-29T11:17:36.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.439+0000 7fa617fff640 10 client.4437.objecter ms_dispatch 0x7fa620000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:36.428 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.439+0000 7fa617fff640 3 client.4437.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:36.429 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.439+0000 7fa617fff640 3 client.4437.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:36.429 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.439+0000 7fa617fff640 20 client.4437.objecter dump_active .. 0 homeless 2024-03-29T11:17:36.429 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.439+0000 7fa61e7fc640 10 monclient: get_auth_request con 0x7fa60006e1f0 auth_method 0 2024-03-29T11:17:36.429 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.439+0000 7fa617fff640 10 client.4437.objecter ms_handle_connect 0x7fa60006e1f0 2024-03-29T11:17:36.429 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.450+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa610050230 con 0x7fa6200a9d70 2024-03-29T11:17:36.429 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.450+0000 7fa617fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:36.429 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.450+0000 7fa617fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:36.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.754+0000 7fa62580e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:36.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.754+0000 7fa62580e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:36.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.754+0000 7fa62580e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:17:36.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.754+0000 7fa62580e640 1 -- 172.21.15.170:0/2943506583 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa604005330 con 0x7fa6200a9d70 2024-03-29T11:17:36.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.755+0000 7fa617fff640 1 -- 172.21.15.170:0/2943506583 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fa610050230 con 0x7fa6200a9d70 2024-03-29T11:17:36.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.755+0000 7fa617fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:36.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.755+0000 7fa617fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:36.732 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:17:36.732 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:36.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.758+0000 7fa62580e640 10 librados: watch_flush enter 2024-03-29T11:17:36.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.758+0000 7fa62580e640 10 librados: watch_flush exit 2024-03-29T11:17:36.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.758+0000 7fa62580e640 20 client.4437.objecter shutdown clearing up homeless session... 2024-03-29T11:17:36.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.758+0000 7fa62580e640 10 client.4437.objecter successfully canceled tick 2024-03-29T11:17:36.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.759+0000 7fa62580e640 1 -- 172.21.15.170:0/2943506583 >> v1:172.21.15.170:6816/3298994444 conn(0x7fa60006e1f0 legacy=0x7fa6000706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:36.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.759+0000 7fa62580e640 10 monclient: shutdown 2024-03-29T11:17:36.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.759+0000 7fa62580e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:36.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.759+0000 7fa62580e640 1 -- 172.21.15.170:0/2943506583 >> v1:172.21.15.170:6789/0 conn(0x7fa6200a9d70 legacy=0x7fa6201deb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:36.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.759+0000 7fa62580e640 1 -- 172.21.15.170:0/2943506583 shutdown_connections 2024-03-29T11:17:36.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.759+0000 7fa62580e640 1 -- 172.21.15.170:0/2943506583 >> 172.21.15.170:0/2943506583 conn(0x7fa620066a00 msgr2=0x7fa620066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:36.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.759+0000 7fa62580e640 1 -- 172.21.15.170:0/2943506583 shutdown_connections 2024-03-29T11:17:36.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.759+0000 7fa62580e640 1 -- 172.21.15.170:0/2943506583 wait complete. 2024-03-29T11:17:36.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:36.759+0000 7fa62580e640 1 librados: shutdown 2024-03-29T11:17:36.750 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:43.558 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:43.690 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.713+0000 7f69e107b640 10 monclient: get_monmap_and_config 2024-03-29T11:17:43.690 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.713+0000 7f69e107b640 10 monclient: build_initial_monmap 2024-03-29T11:17:43.690 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.713+0000 7f69e107b640 10 monclient: monmap: 2024-03-29T11:17:43.691 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:17:43.691 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:43.691 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:17:43.714289+0000 2024-03-29T11:17:43.691 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:17:43.714289+0000 2024-03-29T11:17:43.691 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:43.691 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:43.691 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:17:43.691 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:17:43.691 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:17:43.691 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:43.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.714+0000 7f69e107b640 1 Processor -- start 2024-03-29T11:17:43.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.714+0000 7f69e107b640 1 -- start start 2024-03-29T11:17:43.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.714+0000 7f69e107b640 10 monclient: init 2024-03-29T11:17:43.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.715+0000 7f69e107b640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:43.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.715+0000 7f69e107b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:17:43.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.715+0000 7f69e107b640 10 monclient(hunting): picked mon.noname-a con 0x7f69dc1422b0 addr v1:172.21.15.170:6789/0 2024-03-29T11:17:43.694 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.715+0000 7f69e107b640 10 monclient(hunting): picked mon.noname-b con 0x7f69dc142690 addr v1:172.21.15.170:6790/0 2024-03-29T11:17:43.694 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.715+0000 7f69e107b640 10 monclient(hunting): picked mon.noname-c con 0x7f69dc0b17d0 addr v1:172.21.15.184:6789/0 2024-03-29T11:17:43.694 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.715+0000 7f69e107b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc142c80 con 0x7f69dc1422b0 2024-03-29T11:17:43.694 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.715+0000 7f69e107b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc0aae40 con 0x7f69dc0b17d0 2024-03-29T11:17:43.694 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.715+0000 7f69e107b640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc13fbd0 con 0x7f69dc142690 2024-03-29T11:17:43.694 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.715+0000 7f69e107b640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:43.694 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.715+0000 7f69e107b640 10 monclient(hunting): authenticate will time out at 1245.026921s 2024-03-29T11:17:43.694 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.716+0000 7f69da59c640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f69dc142690 0x7f69dc0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:40054/0 (socket says 172.21.15.170:40054) 2024-03-29T11:17:43.694 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.716+0000 7f69da59c640 1 -- 172.21.15.170:0/3613215639 learned_addr learned my addr 172.21.15.170:0/3613215639 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:17:43.694 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.716+0000 7f69d9d9b640 1 -- 172.21.15.170:0/3613215639 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3030050898 0 0) 0x7f69dc13fbd0 con 0x7f69dc142690 2024-03-29T11:17:43.694 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.716+0000 7f69d9d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:17:43.694 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.716+0000 7f69d9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:43.694 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.716+0000 7f69d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:43.695 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.716+0000 7f69d9d9b640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:17:43.695 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.716+0000 7f69d9d9b640 1 -- 172.21.15.170:0/3613215639 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69c4003520 con 0x7f69dc142690 2024-03-29T11:17:43.695 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.716+0000 7f69d9d9b640 1 -- 172.21.15.170:0/3613215639 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221743251 0 0) 0x7f69dc0aae40 con 0x7f69dc0b17d0 2024-03-29T11:17:43.695 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.716+0000 7f69d9d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:17:43.695 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.716+0000 7f69d9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:43.695 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.716+0000 7f69d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:43.695 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.716+0000 7f69d9d9b640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:17:43.695 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.716+0000 7f69d9d9b640 1 -- 172.21.15.170:0/3613215639 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69dc13fbd0 con 0x7f69dc0b17d0 2024-03-29T11:17:43.695 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.716+0000 7f69d9d9b640 1 -- 172.21.15.170:0/3613215639 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 143003279 0 0) 0x7f69c4003520 con 0x7f69dc142690 2024-03-29T11:17:43.695 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.716+0000 7f69d9d9b640 1 -- 172.21.15.170:0/3613215639 >> v1:172.21.15.184:6789/0 conn(0x7f69dc0b17d0 legacy=0x7f69dc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:43.695 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.716+0000 7f69d9d9b640 1 -- 172.21.15.170:0/3613215639 >> v1:172.21.15.170:6789/0 conn(0x7f69dc1422b0 legacy=0x7f69dc13f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:43.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69d9d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:17:43.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69d9d9b640 1 monclient: found mon.noname-b 2024-03-29T11:17:43.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69d9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:43.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69d9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:17:43.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69d9d9b640 1 -- 172.21.15.170:0/3613215639 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69dc151940 con 0x7f69dc142690 2024-03-29T11:17:43.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69d9d9b640 10 monclient: _finish_auth 0 2024-03-29T11:17:43.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69d9d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:17:43.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69d9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:43.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69d9d9b640 1 -- 172.21.15.170:0/3613215639 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f69cc001690 con 0x7f69dc142690 2024-03-29T11:17:43.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:43.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69d9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:43.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69d9d9b640 10 monclient: dump: 2024-03-29T11:17:43.698 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:43.698 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:43.698 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:43.698 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:43.698 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:43.698 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:43.698 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:43.698 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:43.698 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:43.698 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:43.698 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69e107b640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:17:43.698 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69e107b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:43.698 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69d9d9b640 1 -- 172.21.15.170:0/3613215639 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69cc001870 con 0x7f69dc142690 2024-03-29T11:17:43.698 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69d9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:43.698 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69d9d9b640 1 -- 172.21.15.170:0/3613215639 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f69cc001c00 con 0x7f69dc142690 2024-03-29T11:17:43.699 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69e107b640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:43.699 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69e107b640 4 set_mon_vals no callback set 2024-03-29T11:17:43.699 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:43.699 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69d9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:43.699 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.720+0000 7f69d9d9b640 10 monclient: dump: 2024-03-29T11:17:43.699 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:43.699 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:43.699 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:43.699 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:43.699 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:43.699 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:43.699 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:43.699 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:43.699 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:43.700 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:43.705 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.721+0000 7f69e107b640 10 monclient: shutdown 2024-03-29T11:17:43.705 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.721+0000 7f69e107b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:43.705 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.721+0000 7f69e107b640 1 -- 172.21.15.170:0/3613215639 >> v1:172.21.15.170:6790/0 conn(0x7f69dc142690 legacy=0x7f69dc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:43.705 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.721+0000 7f69e107b640 1 -- 172.21.15.170:0/3613215639 shutdown_connections 2024-03-29T11:17:43.706 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.721+0000 7f69e107b640 1 -- 172.21.15.170:0/3613215639 >> 172.21.15.170:0/3613215639 conn(0x7f69dc066a00 msgr2=0x7f69dc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:43.706 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.721+0000 7f69e107b640 1 -- 172.21.15.170:0/3613215639 shutdown_connections 2024-03-29T11:17:43.706 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.721+0000 7f69e107b640 1 -- 172.21.15.170:0/3613215639 wait complete. 2024-03-29T11:17:43.706 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.723+0000 7f69e107b640 10 monclient: build_initial_monmap 2024-03-29T11:17:43.706 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.723+0000 7f69e107b640 10 monclient: monmap: 2024-03-29T11:17:43.706 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:17:43.706 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:43.706 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:17:43.706 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:17:43.706 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:43.706 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:43.706 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:17:43.706 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:17:43.706 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:17:43.707 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:43.707 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.723+0000 7f69e107b640 1 librados: starting msgr at 2024-03-29T11:17:43.707 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.723+0000 7f69e107b640 1 librados: starting objecter 2024-03-29T11:17:43.707 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.723+0000 7f69e107b640 1 Processor -- start 2024-03-29T11:17:43.707 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.723+0000 7f69e107b640 1 -- start start 2024-03-29T11:17:43.707 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.723+0000 7f69e107b640 1 librados: setting wanted keys 2024-03-29T11:17:43.707 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.723+0000 7f69e107b640 1 librados: calling monclient init 2024-03-29T11:17:43.707 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.723+0000 7f69e107b640 10 monclient: init 2024-03-29T11:17:43.707 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.724+0000 7f69e107b640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:43.707 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.724+0000 7f69e107b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:17:43.707 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.724+0000 7f69e107b640 10 monclient(hunting): picked mon.noname-b con 0x7f69dc0b17d0 addr v1:172.21.15.184:6789/0 2024-03-29T11:17:43.707 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.725+0000 7f69e107b640 10 monclient(hunting): picked mon.noname-c con 0x7f69dc1422b0 addr v1:172.21.15.170:6790/0 2024-03-29T11:17:43.707 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.725+0000 7f69da59c640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f69dc1422b0 0x7f69dc14c540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:41212/0 (socket says 172.21.15.170:41212) 2024-03-29T11:17:43.708 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.725+0000 7f69da59c640 1 -- 172.21.15.170:0/2194481953 learned_addr learned my addr 172.21.15.170:0/2194481953 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:17:43.708 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.725+0000 7f69e107b640 10 monclient(hunting): picked mon.noname-a con 0x7f69dc142690 addr v1:172.21.15.170:6789/0 2024-03-29T11:17:43.708 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.725+0000 7f69e107b640 1 -- 172.21.15.170:0/2194481953 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc14e6c0 con 0x7f69dc142690 2024-03-29T11:17:43.708 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.725+0000 7f69e107b640 1 -- 172.21.15.170:0/2194481953 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc06fd50 con 0x7f69dc0b17d0 2024-03-29T11:17:43.708 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.725+0000 7f69c37fe640 10 client.?.objecter ms_handle_connect 0x7f69dc1422b0 2024-03-29T11:17:43.708 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.725+0000 7f69c37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:43.708 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.725+0000 7f69e107b640 1 -- 172.21.15.170:0/2194481953 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc06eea0 con 0x7f69dc1422b0 2024-03-29T11:17:43.708 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.725+0000 7f69e107b640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:43.708 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.725+0000 7f69e107b640 10 monclient(hunting): authenticate will time out at 1245.036715s 2024-03-29T11:17:43.708 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.725+0000 7f69c37fe640 1 -- 172.21.15.170:0/2194481953 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2697321010 0 0) 0x7f69dc06eea0 con 0x7f69dc1422b0 2024-03-29T11:17:43.708 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.725+0000 7f69c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:43.708 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.725+0000 7f69c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:43.708 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.725+0000 7f69c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:43.709 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.725+0000 7f69c37fe640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:17:43.709 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.725+0000 7f69c37fe640 1 -- 172.21.15.170:0/2194481953 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69c8004eb0 con 0x7f69dc1422b0 2024-03-29T11:17:43.709 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.726+0000 7f69c37fe640 1 -- 172.21.15.170:0/2194481953 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 93501748 0 0) 0x7f69c8004eb0 con 0x7f69dc1422b0 2024-03-29T11:17:43.709 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.726+0000 7f69c37fe640 1 -- 172.21.15.170:0/2194481953 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69dc06eea0 con 0x7f69dc1422b0 2024-03-29T11:17:43.709 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.726+0000 7f69c37fe640 10 client.?.objecter ms_handle_connect 0x7f69dc0b17d0 2024-03-29T11:17:43.709 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.726+0000 7f69c37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:43.709 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.726+0000 7f69c37fe640 1 -- 172.21.15.170:0/2194481953 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f69cc001c80 con 0x7f69dc1422b0 2024-03-29T11:17:43.709 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.726+0000 7f69c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:43.709 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.727+0000 7f69c37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:43.709 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.727+0000 7f69c37fe640 10 monclient(hunting): dump: 2024-03-29T11:17:43.709 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:43.709 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:43.709 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:43.709 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:43.709 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:43.710 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:43.710 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:43.710 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:43.710 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:43.710 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:43.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.727+0000 7f69c37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:43.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.727+0000 7f69c37fe640 1 -- 172.21.15.170:0/2194481953 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549405195 0 0) 0x7f69dc06fd50 con 0x7f69dc0b17d0 2024-03-29T11:17:43.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.727+0000 7f69c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:43.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.727+0000 7f69c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:43.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.727+0000 7f69c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:43.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.727+0000 7f69c37fe640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:17:43.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.727+0000 7f69c37fe640 1 -- 172.21.15.170:0/2194481953 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69c8004eb0 con 0x7f69dc0b17d0 2024-03-29T11:17:43.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.727+0000 7f69c37fe640 1 -- 172.21.15.170:0/2194481953 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 70715323 0 0) 0x7f69dc06eea0 con 0x7f69dc1422b0 2024-03-29T11:17:43.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.727+0000 7f69c37fe640 1 -- 172.21.15.170:0/2194481953 >> v1:172.21.15.184:6789/0 conn(0x7f69dc0b17d0 legacy=0x7f69dc14be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:43.711 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.727+0000 7f69c37fe640 1 -- 172.21.15.170:0/2194481953 >> v1:172.21.15.170:6789/0 conn(0x7f69dc142690 legacy=0x7f69dc1e5f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:43.711 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.728+0000 7f69c37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:17:43.711 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.728+0000 7f69c37fe640 1 monclient: found mon.c 2024-03-29T11:17:43.711 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.728+0000 7f69c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:43.711 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.728+0000 7f69c37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:43.711 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.728+0000 7f69c37fe640 1 -- 172.21.15.170:0/2194481953 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69dc06f030 con 0x7f69dc1422b0 2024-03-29T11:17:43.711 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.728+0000 7f69c37fe640 10 monclient: _finish_auth 0 2024-03-29T11:17:43.711 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.728+0000 7f69c37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:17:43.711 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.728+0000 7f69c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:43.711 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.728+0000 7f69e107b640 5 monclient: authenticate success, global_id 4469 2024-03-29T11:17:43.711 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.728+0000 7f69e107b640 10 monclient: _renew_subs 2024-03-29T11:17:43.711 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.728+0000 7f69e107b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:43.711 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.728+0000 7f69e107b640 1 -- 172.21.15.170:0/2194481953 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69dc1df8c0 con 0x7f69dc1422b0 2024-03-29T11:17:43.711 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.728+0000 7f69e107b640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:43.711 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.728+0000 7f69e107b640 10 monclient: _renew_subs 2024-03-29T11:17:43.712 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.728+0000 7f69e107b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:43.712 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.728+0000 7f69e107b640 1 -- 172.21.15.170:0/2194481953 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69dc1df030 con 0x7f69dc1422b0 2024-03-29T11:17:43.712 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.728+0000 7f69e107b640 1 librados: init done 2024-03-29T11:17:43.712 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.728+0000 7f69c37fe640 1 -- 172.21.15.170:0/2194481953 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69cc004a90 con 0x7f69dc1422b0 2024-03-29T11:17:43.712 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.728+0000 7f69c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:43.712 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.729+0000 7f69d959a640 4 set_mon_vals no callback set 2024-03-29T11:17:43.712 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.729+0000 7f69c37fe640 1 -- 172.21.15.170:0/2194481953 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f69cc004cc0 con 0x7f69dc1422b0 2024-03-29T11:17:43.712 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.729+0000 7f69c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:43.712 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.729+0000 7f69c37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:43.712 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.729+0000 7f69c37fe640 10 monclient: dump: 2024-03-29T11:17:43.712 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:43.712 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:43.712 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:43.712 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:43.712 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:43.713 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:43.713 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:43.713 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:43.713 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:43.713 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:43.715 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.730+0000 7f69e107b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:43.715 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.730+0000 7f69e107b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:43.715 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.730+0000 7f69e107b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:43.715 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.730+0000 7f69e107b640 1 -- 172.21.15.170:0/2194481953 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69dc075f10 con 0x7f69dc1422b0 2024-03-29T11:17:43.715 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.731+0000 7f69c37fe640 1 -- 172.21.15.170:0/2194481953 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f69cc004e20 con 0x7f69dc1422b0 2024-03-29T11:17:43.715 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.731+0000 7f69c37fe640 1 -- 172.21.15.170:0/2194481953 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f69cc084ae0 con 0x7f69dc1422b0 2024-03-29T11:17:43.715 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.731+0000 7f69c37fe640 10 client.4469.objecter ms_dispatch 0x7f69dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:43.715 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.731+0000 7f69c37fe640 3 client.4469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:43.715 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.731+0000 7f69c37fe640 3 client.4469.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:43.715 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.731+0000 7f69c37fe640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-29T11:17:43.715 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.732+0000 7f69dad9d640 10 monclient: get_auth_request con 0x7f69c806ddb0 auth_method 0 2024-03-29T11:17:43.715 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.732+0000 7f69c37fe640 10 client.4469.objecter ms_handle_connect 0x7f69c806ddb0 2024-03-29T11:17:43.715 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.738+0000 7f69c37fe640 1 -- 172.21.15.170:0/2194481953 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f69cc0503c0 con 0x7f69dc1422b0 2024-03-29T11:17:43.716 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.738+0000 7f69c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:43.716 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:43.738+0000 7f69c37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:44.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.044+0000 7f69e107b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:44.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.044+0000 7f69e107b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:44.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.044+0000 7f69e107b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:44.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.044+0000 7f69e107b640 1 -- 172.21.15.170:0/2194481953 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69dc14e4b0 con 0x7f69dc1422b0 2024-03-29T11:17:44.022 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.045+0000 7f69c37fe640 1 -- 172.21.15.170:0/2194481953 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f69cc0522f0 con 0x7f69dc1422b0 2024-03-29T11:17:44.022 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.045+0000 7f69c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:44.022 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.045+0000 7f69c37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:44.024 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:17:44.024 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:44.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.049+0000 7f69e107b640 10 librados: watch_flush enter 2024-03-29T11:17:44.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.049+0000 7f69e107b640 10 librados: watch_flush exit 2024-03-29T11:17:44.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.049+0000 7f69e107b640 20 client.4469.objecter shutdown clearing up homeless session... 2024-03-29T11:17:44.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.049+0000 7f69e107b640 10 client.4469.objecter successfully canceled tick 2024-03-29T11:17:44.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.049+0000 7f69e107b640 1 -- 172.21.15.170:0/2194481953 >> v1:172.21.15.170:6816/3298994444 conn(0x7f69c806ddb0 legacy=0x7f69c8070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:44.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.049+0000 7f69e107b640 10 monclient: shutdown 2024-03-29T11:17:44.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.049+0000 7f69e107b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:44.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.049+0000 7f69e107b640 1 -- 172.21.15.170:0/2194481953 >> v1:172.21.15.170:6790/0 conn(0x7f69dc1422b0 legacy=0x7f69dc14c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:44.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.049+0000 7f69e107b640 1 -- 172.21.15.170:0/2194481953 shutdown_connections 2024-03-29T11:17:44.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.049+0000 7f69e107b640 1 -- 172.21.15.170:0/2194481953 >> 172.21.15.170:0/2194481953 conn(0x7f69dc066a00 msgr2=0x7f69dc13d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:44.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.049+0000 7f69e107b640 1 -- 172.21.15.170:0/2194481953 shutdown_connections 2024-03-29T11:17:44.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.049+0000 7f69e107b640 1 -- 172.21.15.170:0/2194481953 wait complete. 2024-03-29T11:17:44.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:44.049+0000 7f69e107b640 1 librados: shutdown 2024-03-29T11:17:44.041 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:50.848 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:50.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.011+0000 7fbcdf177640 10 monclient: get_monmap_and_config 2024-03-29T11:17:50.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.011+0000 7fbcdf177640 10 monclient: build_initial_monmap 2024-03-29T11:17:50.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.011+0000 7fbcdf177640 10 monclient: monmap: 2024-03-29T11:17:50.988 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:17:50.988 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:50.988 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:17:51.011910+0000 2024-03-29T11:17:50.988 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:17:51.011910+0000 2024-03-29T11:17:50.989 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:50.989 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:50.989 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:17:50.989 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:17:50.989 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:17:50.989 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:50.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.012+0000 7fbcdf177640 1 Processor -- start 2024-03-29T11:17:50.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.012+0000 7fbcdf177640 1 -- start start 2024-03-29T11:17:50.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.012+0000 7fbcdf177640 10 monclient: init 2024-03-29T11:17:50.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.013+0000 7fbcdf177640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:50.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.013+0000 7fbcdf177640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:17:50.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.013+0000 7fbcdf177640 10 monclient(hunting): picked mon.noname-a con 0x7fbcd80b0770 addr v1:172.21.15.170:6789/0 2024-03-29T11:17:50.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.013+0000 7fbcdf177640 10 monclient(hunting): picked mon.noname-b con 0x7fbcd80ad850 addr v1:172.21.15.170:6790/0 2024-03-29T11:17:50.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.013+0000 7fbcdf177640 10 monclient(hunting): picked mon.noname-c con 0x7fbcd813c1b0 addr v1:172.21.15.184:6789/0 2024-03-29T11:17:50.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.013+0000 7fbcdf177640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd80ac8e0 con 0x7fbcd80b0770 2024-03-29T11:17:50.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.013+0000 7fbcdf177640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd80ad330 con 0x7fbcd813c1b0 2024-03-29T11:17:50.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.013+0000 7fbcdf177640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd80acfe0 con 0x7fbcd80ad850 2024-03-29T11:17:50.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.013+0000 7fbcdf177640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:50.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.013+0000 7fbcdf177640 10 monclient(hunting): authenticate will time out at 1252.324161s 2024-03-29T11:17:50.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.013+0000 7fbccffff640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fbcd80ad850 0x7fbcd80aaae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:52038/0 (socket says 172.21.15.170:52038) 2024-03-29T11:17:50.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.013+0000 7fbccffff640 1 -- 172.21.15.170:0/1819342482 learned_addr learned my addr 172.21.15.170:0/1819342482 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:17:50.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.014+0000 7fbccf7fe640 1 -- 172.21.15.170:0/1819342482 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534906065 0 0) 0x7fbcd80acfe0 con 0x7fbcd80ad850 2024-03-29T11:17:50.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.014+0000 7fbccf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:50.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.014+0000 7fbccf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:50.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.014+0000 7fbccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:50.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.014+0000 7fbccf7fe640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:17:50.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.014+0000 7fbccf7fe640 1 -- 172.21.15.170:0/1819342482 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcc0005730 con 0x7fbcd80ad850 2024-03-29T11:17:50.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.014+0000 7fbccf7fe640 1 -- 172.21.15.170:0/1819342482 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42703106 0 0) 0x7fbcd80ad330 con 0x7fbcd813c1b0 2024-03-29T11:17:50.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.014+0000 7fbccf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:50.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.014+0000 7fbccf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:50.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.014+0000 7fbccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:50.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.014+0000 7fbccf7fe640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:17:50.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.014+0000 7fbccf7fe640 1 -- 172.21.15.170:0/1819342482 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcd80acfe0 con 0x7fbcd813c1b0 2024-03-29T11:17:50.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.014+0000 7fbccf7fe640 1 -- 172.21.15.170:0/1819342482 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 333074603 0 0) 0x7fbcc0005730 con 0x7fbcd80ad850 2024-03-29T11:17:50.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.014+0000 7fbccf7fe640 1 -- 172.21.15.170:0/1819342482 >> v1:172.21.15.184:6789/0 conn(0x7fbcd813c1b0 legacy=0x7fbcd814a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:50.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.014+0000 7fbccf7fe640 1 -- 172.21.15.170:0/1819342482 >> v1:172.21.15.170:6789/0 conn(0x7fbcd80b0770 legacy=0x7fbcd8072070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:50.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.016+0000 7fbccf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:17:50.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.016+0000 7fbccf7fe640 1 monclient: found mon.noname-b 2024-03-29T11:17:50.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.016+0000 7fbccf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:50.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.016+0000 7fbccf7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:17:50.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.016+0000 7fbccf7fe640 1 -- 172.21.15.170:0/1819342482 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd80ab1f0 con 0x7fbcd80ad850 2024-03-29T11:17:50.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.016+0000 7fbccf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:17:50.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.016+0000 7fbccf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:17:50.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.016+0000 7fbccf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:50.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.016+0000 7fbccf7fe640 1 -- 172.21.15.170:0/1819342482 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fbcc8001690 con 0x7fbcd80ad850 2024-03-29T11:17:50.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.016+0000 7fbccf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:50.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.016+0000 7fbccf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:50.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.016+0000 7fbccf7fe640 10 monclient: dump: 2024-03-29T11:17:50.995 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:50.995 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:50.995 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:50.995 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:50.995 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:50.995 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:50.995 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:50.996 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:50.996 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:50.996 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:50.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.017+0000 7fbcdf177640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:17:50.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.017+0000 7fbcdf177640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:50.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.017+0000 7fbccf7fe640 1 -- 172.21.15.170:0/1819342482 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbcc80017f0 con 0x7fbcd80ad850 2024-03-29T11:17:50.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.017+0000 7fbccf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:50.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.017+0000 7fbccf7fe640 1 -- 172.21.15.170:0/1819342482 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fbcc8001c00 con 0x7fbcd80ad850 2024-03-29T11:17:50.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.017+0000 7fbcdf177640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:50.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.017+0000 7fbcdf177640 4 set_mon_vals no callback set 2024-03-29T11:17:50.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.017+0000 7fbccf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:50.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.017+0000 7fbccf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:50.997 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.017+0000 7fbccf7fe640 10 monclient: dump: 2024-03-29T11:17:50.997 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:50.997 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:50.997 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:50.997 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:50.997 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:50.997 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:50.997 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:50.997 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:50.997 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:50.997 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:51.004 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.018+0000 7fbcdf177640 10 monclient: shutdown 2024-03-29T11:17:51.005 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.018+0000 7fbcdf177640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:51.005 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.018+0000 7fbcdf177640 1 -- 172.21.15.170:0/1819342482 >> v1:172.21.15.170:6790/0 conn(0x7fbcd80ad850 legacy=0x7fbcd80aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:51.005 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.018+0000 7fbcdf177640 1 -- 172.21.15.170:0/1819342482 shutdown_connections 2024-03-29T11:17:51.005 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.018+0000 7fbcdf177640 1 -- 172.21.15.170:0/1819342482 >> 172.21.15.170:0/1819342482 conn(0x7fbcd806b660 msgr2=0x7fbcd806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:51.005 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.018+0000 7fbcdf177640 1 -- 172.21.15.170:0/1819342482 shutdown_connections 2024-03-29T11:17:51.005 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.018+0000 7fbcdf177640 1 -- 172.21.15.170:0/1819342482 wait complete. 2024-03-29T11:17:51.005 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.019+0000 7fbcdf177640 10 monclient: build_initial_monmap 2024-03-29T11:17:51.005 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.019+0000 7fbcdf177640 10 monclient: monmap: 2024-03-29T11:17:51.005 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:17:51.005 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:51.005 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:17:51.005 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:17:51.006 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:51.006 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:51.006 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:17:51.006 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:17:51.006 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:17:51.006 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:51.006 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.020+0000 7fbcdf177640 1 librados: starting msgr at 2024-03-29T11:17:51.006 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.020+0000 7fbcdf177640 1 librados: starting objecter 2024-03-29T11:17:51.006 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.020+0000 7fbcdf177640 1 Processor -- start 2024-03-29T11:17:51.006 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.020+0000 7fbcdf177640 1 -- start start 2024-03-29T11:17:51.006 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.020+0000 7fbcdf177640 1 librados: setting wanted keys 2024-03-29T11:17:51.006 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.020+0000 7fbcdf177640 1 librados: calling monclient init 2024-03-29T11:17:51.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.020+0000 7fbcdf177640 10 monclient: init 2024-03-29T11:17:51.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.021+0000 7fbcdf177640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:51.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.021+0000 7fbcdf177640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:17:51.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.021+0000 7fbcdf177640 10 monclient(hunting): picked mon.noname-a con 0x7fbcd80ad850 addr v1:172.21.15.170:6789/0 2024-03-29T11:17:51.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.022+0000 7fbcdf177640 10 monclient(hunting): picked mon.noname-b con 0x7fbcd80b0770 addr v1:172.21.15.184:6789/0 2024-03-29T11:17:51.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.022+0000 7fbcdf177640 10 monclient(hunting): picked mon.noname-c con 0x7fbcd813c1b0 addr v1:172.21.15.170:6790/0 2024-03-29T11:17:51.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.022+0000 7fbcdf177640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd8070280 con 0x7fbcd80ad850 2024-03-29T11:17:51.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.022+0000 7fbcdf177640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd81d6420 con 0x7fbcd80b0770 2024-03-29T11:17:51.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.022+0000 7fbcdf177640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd81d7050 con 0x7fbcd813c1b0 2024-03-29T11:17:51.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.022+0000 7fbcdf177640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:51.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.022+0000 7fbcdf177640 10 monclient(hunting): authenticate will time out at 1252.333201s 2024-03-29T11:17:51.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.022+0000 7fbcdd714640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fbcd813c1b0 0x7fbcd81dd5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:54294/0 (socket says 172.21.15.170:54294) 2024-03-29T11:17:51.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.022+0000 7fbcdd714640 1 -- 172.21.15.170:0/3733761579 learned_addr learned my addr 172.21.15.170:0/3733761579 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:17:51.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.022+0000 7fbccd7fa640 10 client.?.objecter ms_handle_connect 0x7fbcd813c1b0 2024-03-29T11:17:51.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.022+0000 7fbccd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:51.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.022+0000 7fbccd7fa640 1 -- 172.21.15.170:0/3733761579 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809453823 0 0) 0x7fbcd81d7050 con 0x7fbcd813c1b0 2024-03-29T11:17:51.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.022+0000 7fbccd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:17:51.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.022+0000 7fbccd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:51.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.022+0000 7fbccd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:51.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.022+0000 7fbccd7fa640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:17:51.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.022+0000 7fbccd7fa640 1 -- 172.21.15.170:0/3733761579 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcc4004eb0 con 0x7fbcd813c1b0 2024-03-29T11:17:51.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.023+0000 7fbccd7fa640 1 -- 172.21.15.170:0/3733761579 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 391576699 0 0) 0x7fbcc4004eb0 con 0x7fbcd813c1b0 2024-03-29T11:17:51.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.023+0000 7fbccd7fa640 1 -- 172.21.15.170:0/3733761579 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbcd81d7050 con 0x7fbcd813c1b0 2024-03-29T11:17:51.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.023+0000 7fbccd7fa640 1 -- 172.21.15.170:0/3733761579 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fbcd0001c80 con 0x7fbcd813c1b0 2024-03-29T11:17:51.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.023+0000 7fbccd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:51.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.023+0000 7fbccd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:51.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.023+0000 7fbccd7fa640 10 monclient(hunting): dump: 2024-03-29T11:17:51.009 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:51.009 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:51.009 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:51.009 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:51.009 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:51.009 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:51.009 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:51.009 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:51.009 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:51.009 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:51.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.023+0000 7fbccd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:51.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.023+0000 7fbccd7fa640 10 client.?.objecter ms_handle_connect 0x7fbcd80b0770 2024-03-29T11:17:51.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.023+0000 7fbccd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:51.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.023+0000 7fbccd7fa640 1 -- 172.21.15.170:0/3733761579 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 756337444 0 0) 0x7fbcd81d7050 con 0x7fbcd813c1b0 2024-03-29T11:17:51.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.023+0000 7fbccd7fa640 1 -- 172.21.15.170:0/3733761579 >> v1:172.21.15.184:6789/0 conn(0x7fbcd80b0770 legacy=0x7fbcd806e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:51.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.023+0000 7fbccd7fa640 1 -- 172.21.15.170:0/3733761579 >> v1:172.21.15.170:6789/0 conn(0x7fbcd80ad850 legacy=0x7fbcd81d6940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:51.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.024+0000 7fbccd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:17:51.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.024+0000 7fbccd7fa640 1 monclient: found mon.c 2024-03-29T11:17:51.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.024+0000 7fbccd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:51.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.024+0000 7fbccd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:51.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.024+0000 7fbccd7fa640 1 -- 172.21.15.170:0/3733761579 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd81d71e0 con 0x7fbcd813c1b0 2024-03-29T11:17:51.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.024+0000 7fbccd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:17:51.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.024+0000 7fbccd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:17:51.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.024+0000 7fbccd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:51.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.024+0000 7fbcdf177640 5 monclient: authenticate success, global_id 4475 2024-03-29T11:17:51.013 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.024+0000 7fbcdf177640 10 monclient: _renew_subs 2024-03-29T11:17:51.013 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.024+0000 7fbcdf177640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:51.013 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.024+0000 7fbcdf177640 1 -- 172.21.15.170:0/3733761579 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd806e980 con 0x7fbcd813c1b0 2024-03-29T11:17:51.013 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.024+0000 7fbcdf177640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:51.013 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.024+0000 7fbcdf177640 10 monclient: _renew_subs 2024-03-29T11:17:51.013 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.024+0000 7fbcdf177640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:51.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.024+0000 7fbcdf177640 1 -- 172.21.15.170:0/3733761579 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd81ded10 con 0x7fbcd813c1b0 2024-03-29T11:17:51.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.024+0000 7fbcdf177640 1 librados: init done 2024-03-29T11:17:51.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.025+0000 7fbccd7fa640 1 -- 172.21.15.170:0/3733761579 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbcd0003970 con 0x7fbcd813c1b0 2024-03-29T11:17:51.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.025+0000 7fbccd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:51.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.025+0000 7fbccd7fa640 1 -- 172.21.15.170:0/3733761579 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fbcd0003dc0 con 0x7fbcd813c1b0 2024-03-29T11:17:51.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.025+0000 7fbccd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:51.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.025+0000 7fbccd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:51.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.025+0000 7fbccd7fa640 10 monclient: dump: 2024-03-29T11:17:51.014 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:51.014 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:51.014 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:51.015 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:51.015 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:51.015 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:51.015 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:51.015 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:51.015 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:51.015 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:51.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.025+0000 7fbcceffd640 4 set_mon_vals no callback set 2024-03-29T11:17:51.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.026+0000 7fbcdf177640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:51.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.027+0000 7fbcdf177640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:51.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.027+0000 7fbcdf177640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:51.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.027+0000 7fbcdf177640 1 -- 172.21.15.170:0/3733761579 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb8005080 con 0x7fbcd813c1b0 2024-03-29T11:17:51.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.027+0000 7fbccd7fa640 1 -- 172.21.15.170:0/3733761579 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fbcd0003fe0 con 0x7fbcd813c1b0 2024-03-29T11:17:51.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.027+0000 7fbccd7fa640 1 -- 172.21.15.170:0/3733761579 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fbcd0084de0 con 0x7fbcd813c1b0 2024-03-29T11:17:51.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.027+0000 7fbccd7fa640 10 client.4475.objecter ms_dispatch 0x7fbcd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:51.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.027+0000 7fbccd7fa640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:51.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.027+0000 7fbccd7fa640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:51.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.028+0000 7fbccd7fa640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-29T11:17:51.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.028+0000 7fbcdcf13640 10 monclient: get_auth_request con 0x7fbcc406db70 auth_method 0 2024-03-29T11:17:51.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.028+0000 7fbccd7fa640 10 client.4475.objecter ms_handle_connect 0x7fbcc406db70 2024-03-29T11:17:51.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.035+0000 7fbccd7fa640 1 -- 172.21.15.170:0/3733761579 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbcd0050720 con 0x7fbcd813c1b0 2024-03-29T11:17:51.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.035+0000 7fbccd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:51.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.035+0000 7fbccd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:51.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.336+0000 7fbcdf177640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:51.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.336+0000 7fbcdf177640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:51.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.336+0000 7fbcdf177640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:51.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.336+0000 7fbcdf177640 1 -- 172.21.15.170:0/3733761579 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbcb8005330 con 0x7fbcd813c1b0 2024-03-29T11:17:51.315 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.338+0000 7fbccd7fa640 1 -- 172.21.15.170:0/3733761579 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fbcd0084490 con 0x7fbcd813c1b0 2024-03-29T11:17:51.315 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.338+0000 7fbccd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:51.315 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.338+0000 7fbccd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:51.315 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:17:51.315 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:51.318 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.341+0000 7fbcdf177640 10 librados: watch_flush enter 2024-03-29T11:17:51.318 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.341+0000 7fbcdf177640 10 librados: watch_flush exit 2024-03-29T11:17:51.318 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.341+0000 7fbcdf177640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-29T11:17:51.318 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.341+0000 7fbcdf177640 10 client.4475.objecter successfully canceled tick 2024-03-29T11:17:51.319 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.341+0000 7fbcdf177640 1 -- 172.21.15.170:0/3733761579 >> v1:172.21.15.170:6816/3298994444 conn(0x7fbcc406db70 legacy=0x7fbcc4070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:51.319 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.341+0000 7fbcdf177640 10 monclient: shutdown 2024-03-29T11:17:51.319 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.342+0000 7fbcdf177640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:51.319 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.342+0000 7fbcdf177640 1 -- 172.21.15.170:0/3733761579 >> v1:172.21.15.170:6790/0 conn(0x7fbcd813c1b0 legacy=0x7fbcd81dd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:51.319 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.342+0000 7fbcdf177640 1 -- 172.21.15.170:0/3733761579 shutdown_connections 2024-03-29T11:17:51.319 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.342+0000 7fbcdf177640 1 -- 172.21.15.170:0/3733761579 >> 172.21.15.170:0/3733761579 conn(0x7fbcd806b660 msgr2=0x7fbcd80adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:51.319 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.342+0000 7fbcdf177640 1 -- 172.21.15.170:0/3733761579 shutdown_connections 2024-03-29T11:17:51.320 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.342+0000 7fbcdf177640 1 -- 172.21.15.170:0/3733761579 wait complete. 2024-03-29T11:17:51.320 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:51.342+0000 7fbcdf177640 1 librados: shutdown 2024-03-29T11:17:51.334 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:58.141 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:58.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.319+0000 7fd047519640 10 monclient: get_monmap_and_config 2024-03-29T11:17:58.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.319+0000 7fd047519640 10 monclient: build_initial_monmap 2024-03-29T11:17:58.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.319+0000 7fd047519640 10 monclient: monmap: 2024-03-29T11:17:58.297 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:17:58.297 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:58.297 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:17:58.321077+0000 2024-03-29T11:17:58.297 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:17:58.321077+0000 2024-03-29T11:17:58.297 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:58.297 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:58.298 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:17:58.298 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:17:58.298 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:17:58.298 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:58.298 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.320+0000 7fd047519640 1 Processor -- start 2024-03-29T11:17:58.298 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.320+0000 7fd047519640 1 -- start start 2024-03-29T11:17:58.298 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.320+0000 7fd047519640 10 monclient: init 2024-03-29T11:17:58.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.321+0000 7fd047519640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:58.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.321+0000 7fd047519640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:17:58.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.322+0000 7fd047519640 10 monclient(hunting): picked mon.noname-b con 0x7fd0380389e0 addr v1:172.21.15.170:6790/0 2024-03-29T11:17:58.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.322+0000 7fd047519640 10 monclient(hunting): picked mon.noname-c con 0x7fd038038dc0 addr v1:172.21.15.184:6789/0 2024-03-29T11:17:58.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.322+0000 7fd047519640 10 monclient(hunting): picked mon.noname-a con 0x7fd038030660 addr v1:172.21.15.170:6789/0 2024-03-29T11:17:58.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.322+0000 7fd047519640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0380c54c0 con 0x7fd038030660 2024-03-29T11:17:58.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.322+0000 7fd047519640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd038036430 con 0x7fd038038dc0 2024-03-29T11:17:58.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.322+0000 7fd047519640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd038039840 con 0x7fd0380389e0 2024-03-29T11:17:58.301 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.322+0000 7fd047519640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:58.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.322+0000 7fd047519640 10 monclient(hunting): authenticate will time out at 1259.634084s 2024-03-29T11:17:58.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.325+0000 7fd046517640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fd0380389e0 0x7fd038033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:38839/0 (socket says 172.21.15.170:38839) 2024-03-29T11:17:58.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.325+0000 7fd046517640 1 -- 172.21.15.170:0/811198360 learned_addr learned my addr 172.21.15.170:0/811198360 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:17:58.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.326+0000 7fd045515640 1 -- 172.21.15.170:0/811198360 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577525671 0 0) 0x7fd038039840 con 0x7fd0380389e0 2024-03-29T11:17:58.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.327+0000 7fd045515640 10 monclient(hunting): _negotiate 2024-03-29T11:17:58.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.327+0000 7fd045515640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:58.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.327+0000 7fd045515640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:58.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.327+0000 7fd045515640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:17:58.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.327+0000 7fd045515640 1 -- 172.21.15.170:0/811198360 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd02c003520 con 0x7fd0380389e0 2024-03-29T11:17:58.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.327+0000 7fd045515640 1 -- 172.21.15.170:0/811198360 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006710973 0 0) 0x7fd038036430 con 0x7fd038038dc0 2024-03-29T11:17:58.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.327+0000 7fd045515640 10 monclient(hunting): _negotiate 2024-03-29T11:17:58.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.327+0000 7fd045515640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:58.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.327+0000 7fd045515640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:58.306 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.327+0000 7fd045515640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:17:58.306 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.327+0000 7fd045515640 1 -- 172.21.15.170:0/811198360 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd02c0048a0 con 0x7fd038038dc0 2024-03-29T11:17:58.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.329+0000 7fd045515640 1 -- 172.21.15.170:0/811198360 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1327775613 0 0) 0x7fd02c003520 con 0x7fd0380389e0 2024-03-29T11:17:58.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.329+0000 7fd045515640 1 -- 172.21.15.170:0/811198360 >> v1:172.21.15.184:6789/0 conn(0x7fd038038dc0 legacy=0x7fd0380cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:58.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.329+0000 7fd045515640 1 -- 172.21.15.170:0/811198360 >> v1:172.21.15.170:6789/0 conn(0x7fd038030660 legacy=0x7fd0380d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:58.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.329+0000 7fd045515640 10 monclient: _finish_hunting 0 2024-03-29T11:17:58.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.329+0000 7fd045515640 1 monclient: found mon.noname-b 2024-03-29T11:17:58.307 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.329+0000 7fd045515640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:58.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.329+0000 7fd045515640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:17:58.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.330+0000 7fd045515640 1 -- 172.21.15.170:0/811198360 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd038035690 con 0x7fd0380389e0 2024-03-29T11:17:58.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.330+0000 7fd045515640 10 monclient: _finish_auth 0 2024-03-29T11:17:58.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.330+0000 7fd045515640 10 monclient: _check_auth_tickets 2024-03-29T11:17:58.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.330+0000 7fd045515640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:58.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.330+0000 7fd045515640 1 -- 172.21.15.170:0/811198360 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fd030001670 con 0x7fd0380389e0 2024-03-29T11:17:58.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.330+0000 7fd045515640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:58.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.330+0000 7fd045515640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:58.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.330+0000 7fd045515640 10 monclient: dump: 2024-03-29T11:17:58.308 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:58.308 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:58.308 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:58.308 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:58.309 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:58.309 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:58.309 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:58.309 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:58.309 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:58.309 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:58.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.330+0000 7fd047519640 5 monclient: authenticate success, global_id 4478 2024-03-29T11:17:58.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.330+0000 7fd047519640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:58.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.331+0000 7fd045515640 1 -- 172.21.15.170:0/811198360 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0300017d0 con 0x7fd0380389e0 2024-03-29T11:17:58.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.331+0000 7fd045515640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:58.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.331+0000 7fd045515640 1 -- 172.21.15.170:0/811198360 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fd030001be0 con 0x7fd0380389e0 2024-03-29T11:17:58.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.331+0000 7fd045515640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:58.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.331+0000 7fd045515640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:58.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.331+0000 7fd045515640 10 monclient: dump: 2024-03-29T11:17:58.310 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:58.311 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:58.311 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:58.311 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:58.311 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:58.311 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:58.311 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:58.311 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:58.311 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:58.311 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:58.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.331+0000 7fd047519640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:58.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.331+0000 7fd047519640 4 set_mon_vals no callback set 2024-03-29T11:17:58.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.332+0000 7fd047519640 10 monclient: shutdown 2024-03-29T11:17:58.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.332+0000 7fd047519640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:58.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.332+0000 7fd047519640 1 -- 172.21.15.170:0/811198360 >> v1:172.21.15.170:6790/0 conn(0x7fd0380389e0 legacy=0x7fd038033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:58.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.333+0000 7fd047519640 1 -- 172.21.15.170:0/811198360 shutdown_connections 2024-03-29T11:17:58.312 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.333+0000 7fd047519640 1 -- 172.21.15.170:0/811198360 >> 172.21.15.170:0/811198360 conn(0x7fd038005c00 msgr2=0x7fd038006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:58.312 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.333+0000 7fd047519640 1 -- 172.21.15.170:0/811198360 shutdown_connections 2024-03-29T11:17:58.312 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.333+0000 7fd047519640 1 -- 172.21.15.170:0/811198360 wait complete. 2024-03-29T11:17:58.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.334+0000 7fd047519640 10 monclient: build_initial_monmap 2024-03-29T11:17:58.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.334+0000 7fd047519640 10 monclient: monmap: 2024-03-29T11:17:58.313 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:17:58.313 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:58.313 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:17:58.313 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:17:58.313 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:58.313 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:58.313 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:17:58.313 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:17:58.313 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:17:58.313 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:58.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.335+0000 7fd047519640 1 librados: starting msgr at 2024-03-29T11:17:58.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.335+0000 7fd047519640 1 librados: starting objecter 2024-03-29T11:17:58.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.335+0000 7fd047519640 1 Processor -- start 2024-03-29T11:17:58.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.335+0000 7fd047519640 1 -- start start 2024-03-29T11:17:58.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.335+0000 7fd047519640 1 librados: setting wanted keys 2024-03-29T11:17:58.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.335+0000 7fd047519640 1 librados: calling monclient init 2024-03-29T11:17:58.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.335+0000 7fd047519640 10 monclient: init 2024-03-29T11:17:58.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.336+0000 7fd047519640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:58.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.336+0000 7fd047519640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:58.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.336+0000 7fd047519640 10 monclient(hunting): picked mon.noname-b con 0x7fd038030660 addr v1:172.21.15.184:6789/0 2024-03-29T11:17:58.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.336+0000 7fd047519640 10 monclient(hunting): picked mon.noname-a con 0x7fd0380389e0 addr v1:172.21.15.170:6789/0 2024-03-29T11:17:58.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.336+0000 7fd047519640 10 monclient(hunting): picked mon.noname-c con 0x7fd03802fc60 addr v1:172.21.15.170:6790/0 2024-03-29T11:17:58.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.336+0000 7fd047519640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd03802f120 con 0x7fd0380389e0 2024-03-29T11:17:58.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.336+0000 7fd047519640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd03802da80 con 0x7fd038030660 2024-03-29T11:17:58.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.336+0000 7fd047519640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd03802ed00 con 0x7fd03802fc60 2024-03-29T11:17:58.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.336+0000 7fd047519640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:58.317 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.336+0000 7fd047519640 10 monclient(hunting): authenticate will time out at 1259.648437s 2024-03-29T11:17:58.318 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.341+0000 7fd045d16640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fd0380389e0 0x7fd03802f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:47229/0 (socket says 172.21.15.170:47229) 2024-03-29T11:17:58.318 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.341+0000 7fd045d16640 1 -- 172.21.15.170:0/1895466351 learned_addr learned my addr 172.21.15.170:0/1895466351 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:17:58.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.345+0000 7fd036ffd640 10 client.?.objecter ms_handle_connect 0x7fd0380389e0 2024-03-29T11:17:58.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.345+0000 7fd036ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:58.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.345+0000 7fd036ffd640 10 client.?.objecter ms_handle_connect 0x7fd03802fc60 2024-03-29T11:17:58.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.345+0000 7fd036ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:58.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.345+0000 7fd036ffd640 10 client.?.objecter ms_handle_connect 0x7fd038030660 2024-03-29T11:17:58.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.345+0000 7fd036ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:58.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.345+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751456260 0 0) 0x7fd03802f120 con 0x7fd0380389e0 2024-03-29T11:17:58.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.345+0000 7fd036ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:58.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.345+0000 7fd036ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:58.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.345+0000 7fd036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:58.324 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:17:58.324 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd024004eb0 con 0x7fd0380389e0 2024-03-29T11:17:58.324 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069624413 0 0) 0x7fd03802ed00 con 0x7fd03802fc60 2024-03-29T11:17:58.324 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:58.324 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:58.324 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:58.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:17:58.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd03802f120 con 0x7fd03802fc60 2024-03-29T11:17:58.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388861302 0 0) 0x7fd03802da80 con 0x7fd038030660 2024-03-29T11:17:58.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:58.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:58.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:58.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:17:58.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd03802ed00 con 0x7fd038030660 2024-03-29T11:17:58.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2912854957 0 0) 0x7fd03802f120 con 0x7fd03802fc60 2024-03-29T11:17:58.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd03802da80 con 0x7fd03802fc60 2024-03-29T11:17:58.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fd04004fdd0 con 0x7fd03802fc60 2024-03-29T11:17:58.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:58.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:58.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 10 monclient(hunting): dump: 2024-03-29T11:17:58.326 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:58.326 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:58.326 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:58.326 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:58.326 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:58.326 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:58.326 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:58.327 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:58.327 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:58.327 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:58.327 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:58.327 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3588351489 0 0) 0x7fd03802ed00 con 0x7fd038030660 2024-03-29T11:17:58.327 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd03802f120 con 0x7fd038030660 2024-03-29T11:17:58.327 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fd030001ae0 con 0x7fd038030660 2024-03-29T11:17:58.327 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:58.327 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:58.327 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 10 monclient(hunting): dump: 2024-03-29T11:17:58.327 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:58.327 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:58.327 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:58.327 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:58.327 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:58.328 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:58.328 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:58.328 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:58.328 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:58.328 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:58.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1294381823 0 0) 0x7fd03802da80 con 0x7fd03802fc60 2024-03-29T11:17:58.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 >> v1:172.21.15.184:6789/0 conn(0x7fd038030660 legacy=0x7fd0380c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:58.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.347+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 >> v1:172.21.15.170:6789/0 conn(0x7fd0380389e0 legacy=0x7fd03802f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:58.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.348+0000 7fd036ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:17:58.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.348+0000 7fd036ffd640 1 monclient: found mon.c 2024-03-29T11:17:58.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.348+0000 7fd036ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:58.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.348+0000 7fd036ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:58.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.348+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0380300c0 con 0x7fd03802fc60 2024-03-29T11:17:58.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.348+0000 7fd036ffd640 10 monclient: _finish_auth 0 2024-03-29T11:17:58.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.348+0000 7fd036ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:17:58.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.348+0000 7fd036ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:58.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.348+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd04004f400 con 0x7fd03802fc60 2024-03-29T11:17:58.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.348+0000 7fd036ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:58.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.348+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fd040075f10 con 0x7fd03802fc60 2024-03-29T11:17:58.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.348+0000 7fd036ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:58.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.348+0000 7fd036ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:58.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.348+0000 7fd036ffd640 10 monclient: dump: 2024-03-29T11:17:58.329 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:17:58.329 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:17:58.329 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:58.329 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:17:58.329 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:58.329 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:17:58.329 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:17:58.329 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:17:58.329 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:17:58.330 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:17:58.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.350+0000 7fd047519640 5 monclient: authenticate success, global_id 4481 2024-03-29T11:17:58.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.350+0000 7fd047519640 10 monclient: _renew_subs 2024-03-29T11:17:58.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.350+0000 7fd047519640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:58.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.350+0000 7fd047519640 1 -- 172.21.15.170:0/1895466351 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03815a5a0 con 0x7fd03802fc60 2024-03-29T11:17:58.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.350+0000 7fd047519640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:58.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.350+0000 7fd047519640 10 monclient: _renew_subs 2024-03-29T11:17:58.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.350+0000 7fd047519640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:58.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.350+0000 7fd047519640 1 -- 172.21.15.170:0/1895466351 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd03802d810 con 0x7fd03802fc60 2024-03-29T11:17:58.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.350+0000 7fd047519640 1 librados: init done 2024-03-29T11:17:58.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.350+0000 7fd044d14640 4 set_mon_vals no callback set 2024-03-29T11:17:58.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.352+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fd040067bf0 con 0x7fd03802fc60 2024-03-29T11:17:58.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.353+0000 7fd047519640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:58.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.353+0000 7fd047519640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:58.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.353+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fd0400fe1a0 con 0x7fd03802fc60 2024-03-29T11:17:58.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.353+0000 7fd036ffd640 10 client.4481.objecter ms_dispatch 0x7fd040000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:58.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.353+0000 7fd036ffd640 3 client.4481.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:58.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.353+0000 7fd036ffd640 3 client.4481.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:58.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.353+0000 7fd036ffd640 20 client.4481.objecter dump_active .. 0 homeless 2024-03-29T11:17:58.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.353+0000 7fd047519640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:58.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.353+0000 7fd047519640 1 -- 172.21.15.170:0/1895466351 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd028005080 con 0x7fd03802fc60 2024-03-29T11:17:58.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.354+0000 7fd046517640 10 monclient: get_auth_request con 0x7fd02406dde0 auth_method 0 2024-03-29T11:17:58.333 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.354+0000 7fd036ffd640 10 client.4481.objecter ms_handle_connect 0x7fd02406dde0 2024-03-29T11:17:58.339 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.362+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd0400c9ab0 con 0x7fd03802fc60 2024-03-29T11:17:58.339 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.362+0000 7fd036ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:58.340 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.362+0000 7fd036ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:58.667 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.688+0000 7fd047519640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:58.667 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.688+0000 7fd047519640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:58.667 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.688+0000 7fd047519640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:17:58.667 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.688+0000 7fd047519640 1 -- 172.21.15.170:0/1895466351 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd028005330 con 0x7fd03802fc60 2024-03-29T11:17:58.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.692+0000 7fd036ffd640 1 -- 172.21.15.170:0/1895466351 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fd0400cb9e0 con 0x7fd03802fc60 2024-03-29T11:17:58.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.692+0000 7fd036ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:58.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.692+0000 7fd036ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:58.672 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:17:58.672 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:58.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.698+0000 7fd047519640 10 librados: watch_flush enter 2024-03-29T11:17:58.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.698+0000 7fd047519640 10 librados: watch_flush exit 2024-03-29T11:17:58.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.698+0000 7fd047519640 20 client.4481.objecter shutdown clearing up homeless session... 2024-03-29T11:17:58.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.698+0000 7fd047519640 10 client.4481.objecter successfully canceled tick 2024-03-29T11:17:58.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.698+0000 7fd047519640 1 -- 172.21.15.170:0/1895466351 >> v1:172.21.15.170:6816/3298994444 conn(0x7fd02406dde0 legacy=0x7fd0240702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:58.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.698+0000 7fd047519640 10 monclient: shutdown 2024-03-29T11:17:58.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.698+0000 7fd047519640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:58.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.698+0000 7fd047519640 1 -- 172.21.15.170:0/1895466351 >> v1:172.21.15.170:6790/0 conn(0x7fd03802fc60 legacy=0x7fd03815cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:58.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.698+0000 7fd047519640 1 -- 172.21.15.170:0/1895466351 shutdown_connections 2024-03-29T11:17:58.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.698+0000 7fd047519640 1 -- 172.21.15.170:0/1895466351 >> 172.21.15.170:0/1895466351 conn(0x7fd038005c00 msgr2=0x7fd038034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:58.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.698+0000 7fd047519640 1 -- 172.21.15.170:0/1895466351 shutdown_connections 2024-03-29T11:17:58.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.698+0000 7fd047519640 1 -- 172.21.15.170:0/1895466351 wait complete. 2024-03-29T11:17:58.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:17:58.698+0000 7fd047519640 1 librados: shutdown 2024-03-29T11:17:58.696 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:05.503 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:05.657 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.678+0000 7f32abb7b640 10 monclient: get_monmap_and_config 2024-03-29T11:18:05.657 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.678+0000 7f32abb7b640 10 monclient: build_initial_monmap 2024-03-29T11:18:05.657 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.678+0000 7f32abb7b640 10 monclient: monmap: 2024-03-29T11:18:05.657 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:18:05.657 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:05.658 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:18:05.680228+0000 2024-03-29T11:18:05.658 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:18:05.680228+0000 2024-03-29T11:18:05.658 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:05.658 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:05.658 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:18:05.658 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:18:05.658 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:18:05.658 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:05.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.680+0000 7f32abb7b640 1 Processor -- start 2024-03-29T11:18:05.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.681+0000 7f32abb7b640 1 -- start start 2024-03-29T11:18:05.659 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.681+0000 7f32abb7b640 10 monclient: init 2024-03-29T11:18:05.660 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.682+0000 7f32abb7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:05.660 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.682+0000 7f32abb7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:05.660 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.682+0000 7f32abb7b640 10 monclient(hunting): picked mon.noname-a con 0x7f32a40b4100 addr v1:172.21.15.170:6789/0 2024-03-29T11:18:05.660 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.682+0000 7f32abb7b640 10 monclient(hunting): picked mon.noname-c con 0x7f32a413ee90 addr v1:172.21.15.184:6789/0 2024-03-29T11:18:05.660 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.682+0000 7f32abb7b640 10 monclient(hunting): picked mon.noname-b con 0x7f32a40af900 addr v1:172.21.15.170:6790/0 2024-03-29T11:18:05.660 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.682+0000 7f32abb7b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a40afd10 con 0x7f32a40b4100 2024-03-29T11:18:05.661 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.682+0000 7f32abb7b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a40af6f0 con 0x7f32a413ee90 2024-03-29T11:18:05.661 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.682+0000 7f32abb7b640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a40aaf60 con 0x7f32a40af900 2024-03-29T11:18:05.661 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.682+0000 7f32abb7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:05.661 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.682+0000 7f32abb7b640 10 monclient(hunting): authenticate will time out at 1266.994442s 2024-03-29T11:18:05.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.688+0000 7f32a9917640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f32a40b4100 0x7f32a40a9850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:49277/0 (socket says 172.21.15.170:49277) 2024-03-29T11:18:05.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.688+0000 7f32a9917640 1 -- 172.21.15.170:0/3663357737 learned_addr learned my addr 172.21.15.170:0/3663357737 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:18:05.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.688+0000 7f32a8915640 1 -- 172.21.15.170:0/3663357737 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378351152 0 0) 0x7f32a40afd10 con 0x7f32a40b4100 2024-03-29T11:18:05.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.688+0000 7f32a8915640 10 monclient(hunting): _negotiate 2024-03-29T11:18:05.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.688+0000 7f32a8915640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:05.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.688+0000 7f32a8915640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:05.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.688+0000 7f32a8915640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:18:05.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.688+0000 7f32a8915640 1 -- 172.21.15.170:0/3663357737 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3294003520 con 0x7f32a40b4100 2024-03-29T11:18:05.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.688+0000 7f32a8915640 1 -- 172.21.15.170:0/3663357737 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2485336291 0 0) 0x7f3294003520 con 0x7f32a40b4100 2024-03-29T11:18:05.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.688+0000 7f32a8915640 1 -- 172.21.15.170:0/3663357737 >> v1:172.21.15.170:6790/0 conn(0x7f32a40af900 legacy=0x7f32a413b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:05.670 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32a8915640 1 -- 172.21.15.170:0/3663357737 >> v1:172.21.15.184:6789/0 conn(0x7f32a413ee90 legacy=0x7f32a40adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:05.670 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32a8915640 10 monclient: _finish_hunting 0 2024-03-29T11:18:05.670 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32a8915640 1 monclient: found mon.noname-a 2024-03-29T11:18:05.670 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32a8915640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:05.670 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32a8915640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:18:05.670 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32a8915640 1 -- 172.21.15.170:0/3663357737 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a413e970 con 0x7f32a40b4100 2024-03-29T11:18:05.671 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32a8915640 10 monclient: _finish_auth 0 2024-03-29T11:18:05.671 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32a8915640 10 monclient: _check_auth_tickets 2024-03-29T11:18:05.671 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32a8915640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:05.671 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32abb7b640 5 monclient: authenticate success, global_id 4446 2024-03-29T11:18:05.671 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32abb7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:05.671 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32a8915640 1 -- 172.21.15.170:0/3663357737 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f329c001690 con 0x7f32a40b4100 2024-03-29T11:18:05.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32a8915640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:05.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32a8915640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:05.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32a8915640 10 monclient: dump: 2024-03-29T11:18:05.672 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:05.672 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:05.672 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:05.672 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:05.672 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:05.672 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:05.672 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:05.672 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:05.672 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:05.672 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:05.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32abb7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:05.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32a8915640 1 -- 172.21.15.170:0/3663357737 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f329c0019a0 con 0x7f32a40b4100 2024-03-29T11:18:05.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32a8915640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:05.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32abb7b640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:05.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.691+0000 7f32abb7b640 4 set_mon_vals no callback set 2024-03-29T11:18:05.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.692+0000 7f32abb7b640 10 monclient: shutdown 2024-03-29T11:18:05.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.692+0000 7f32abb7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:05.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.692+0000 7f32abb7b640 1 -- 172.21.15.170:0/3663357737 >> v1:172.21.15.170:6789/0 conn(0x7f32a40b4100 legacy=0x7f32a40a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:05.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.692+0000 7f32abb7b640 1 -- 172.21.15.170:0/3663357737 shutdown_connections 2024-03-29T11:18:05.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.692+0000 7f32abb7b640 1 -- 172.21.15.170:0/3663357737 >> 172.21.15.170:0/3663357737 conn(0x7f32a40b35e0 msgr2=0x7f32a40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:05.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.693+0000 7f32abb7b640 1 -- 172.21.15.170:0/3663357737 shutdown_connections 2024-03-29T11:18:05.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.693+0000 7f32abb7b640 1 -- 172.21.15.170:0/3663357737 wait complete. 2024-03-29T11:18:05.674 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.694+0000 7f32abb7b640 10 monclient: build_initial_monmap 2024-03-29T11:18:05.674 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.694+0000 7f32abb7b640 10 monclient: monmap: 2024-03-29T11:18:05.674 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:18:05.674 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:05.674 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:18:05.674 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:18:05.674 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:05.674 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:05.674 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:18:05.674 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:18:05.675 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:18:05.675 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:05.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.695+0000 7f32abb7b640 1 librados: starting msgr at 2024-03-29T11:18:05.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.695+0000 7f32abb7b640 1 librados: starting objecter 2024-03-29T11:18:05.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.695+0000 7f32abb7b640 1 Processor -- start 2024-03-29T11:18:05.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.695+0000 7f32abb7b640 1 -- start start 2024-03-29T11:18:05.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.695+0000 7f32abb7b640 1 librados: setting wanted keys 2024-03-29T11:18:05.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.695+0000 7f32abb7b640 1 librados: calling monclient init 2024-03-29T11:18:05.675 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.695+0000 7f32abb7b640 10 monclient: init 2024-03-29T11:18:05.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.695+0000 7f32abb7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:05.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.696+0000 7f32abb7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:05.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.696+0000 7f32abb7b640 10 monclient(hunting): picked mon.noname-c con 0x7f32a40af900 addr v1:172.21.15.170:6790/0 2024-03-29T11:18:05.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.696+0000 7f32abb7b640 10 monclient(hunting): picked mon.noname-b con 0x7f32a40b4100 addr v1:172.21.15.184:6789/0 2024-03-29T11:18:05.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.696+0000 7f32abb7b640 10 monclient(hunting): picked mon.noname-a con 0x7f32a413ee90 addr v1:172.21.15.170:6789/0 2024-03-29T11:18:05.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.696+0000 7f32abb7b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a412c220 con 0x7f32a413ee90 2024-03-29T11:18:05.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.696+0000 7f32abb7b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a412a1c0 con 0x7f32a40b4100 2024-03-29T11:18:05.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.696+0000 7f32abb7b640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a412c490 con 0x7f32a40af900 2024-03-29T11:18:05.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.696+0000 7f32abb7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:05.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.696+0000 7f32abb7b640 10 monclient(hunting): authenticate will time out at 1267.008103s 2024-03-29T11:18:05.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.703+0000 7f32a9917640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f32a40af900 0x7f32a41293a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:33843/0 (socket says 172.21.15.170:33843) 2024-03-29T11:18:05.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.703+0000 7f32a9917640 1 -- 172.21.15.170:0/1916233038 learned_addr learned my addr 172.21.15.170:0/1916233038 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:18:05.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.704+0000 7f32a27fc640 10 client.?.objecter ms_handle_connect 0x7f32a40af900 2024-03-29T11:18:05.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.704+0000 7f32a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:05.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.705+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1902618085 0 0) 0x7f32a412c490 con 0x7f32a40af900 2024-03-29T11:18:05.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.705+0000 7f32a27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:05.683 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.705+0000 7f32a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:05.683 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.705+0000 7f32a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:05.683 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.705+0000 7f32a27fc640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:18:05.683 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.705+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3288004eb0 con 0x7f32a40af900 2024-03-29T11:18:05.692 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.708+0000 7f32a27fc640 10 client.?.objecter ms_handle_connect 0x7f32a40b4100 2024-03-29T11:18:05.692 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.708+0000 7f32a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:05.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.708+0000 7f32a27fc640 10 client.?.objecter ms_handle_connect 0x7f32a413ee90 2024-03-29T11:18:05.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.708+0000 7f32a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:05.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.708+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461610933 0 0) 0x7f32a412a1c0 con 0x7f32a40b4100 2024-03-29T11:18:05.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.708+0000 7f32a27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:05.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.708+0000 7f32a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:05.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.708+0000 7f32a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:05.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.708+0000 7f32a27fc640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:18:05.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.708+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32880051d0 con 0x7f32a40b4100 2024-03-29T11:18:05.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.708+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3539749745 0 0) 0x7f3288004eb0 con 0x7f32a40af900 2024-03-29T11:18:05.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.708+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32a412a1c0 con 0x7f32a40af900 2024-03-29T11:18:05.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.708+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f329c0047e0 con 0x7f32a40af900 2024-03-29T11:18:05.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.708+0000 7f32a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:05.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.708+0000 7f32a27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:05.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.708+0000 7f32a27fc640 10 monclient(hunting): dump: 2024-03-29T11:18:05.694 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:05.694 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:05.694 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:05.694 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:05.694 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:05.694 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:05.694 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:05.694 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:05.694 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:05.694 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:05.694 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.708+0000 7f32a27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:05.694 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.709+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1392674072 0 0) 0x7f32880051d0 con 0x7f32a40b4100 2024-03-29T11:18:05.694 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.709+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3288004eb0 con 0x7f32a40b4100 2024-03-29T11:18:05.695 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.709+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3290001790 con 0x7f32a40b4100 2024-03-29T11:18:05.695 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.709+0000 7f32a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:05.695 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.709+0000 7f32a27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:05.695 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.709+0000 7f32a27fc640 10 monclient(hunting): dump: 2024-03-29T11:18:05.695 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:05.695 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:05.695 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:05.695 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:05.695 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:05.695 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:05.695 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:05.695 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:05.695 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:05.696 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:05.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.709+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081023074 0 0) 0x7f32a412c220 con 0x7f32a413ee90 2024-03-29T11:18:05.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.709+0000 7f32a27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:05.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.709+0000 7f32a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:05.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.709+0000 7f32a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:05.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.709+0000 7f32a27fc640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:18:05.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.709+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32880051d0 con 0x7f32a413ee90 2024-03-29T11:18:05.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1559258729 0 0) 0x7f32880051d0 con 0x7f32a413ee90 2024-03-29T11:18:05.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32a412c220 con 0x7f32a413ee90 2024-03-29T11:18:05.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4139504583 0 0) 0x7f3288004eb0 con 0x7f32a40b4100 2024-03-29T11:18:05.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 >> v1:172.21.15.170:6790/0 conn(0x7f32a40af900 legacy=0x7f32a41293a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:05.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 >> v1:172.21.15.170:6789/0 conn(0x7f32a413ee90 legacy=0x7f32a4129ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:05.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32a27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:05.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32a27fc640 1 monclient: found mon.b 2024-03-29T11:18:05.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:05.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32a27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:18:05.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a412c770 con 0x7f32a40b4100 2024-03-29T11:18:05.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32a27fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:05.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32a27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:05.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:05.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32abb7b640 5 monclient: authenticate success, global_id 4447 2024-03-29T11:18:05.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32abb7b640 10 monclient: _renew_subs 2024-03-29T11:18:05.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32abb7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:18:05.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32abb7b640 1 -- 172.21.15.170:0/1916233038 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a412ca50 con 0x7f32a40b4100 2024-03-29T11:18:05.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32abb7b640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:05.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32abb7b640 10 monclient: _renew_subs 2024-03-29T11:18:05.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32abb7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:18:05.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32abb7b640 1 -- 172.21.15.170:0/1916233038 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a4142510 con 0x7f32a40b4100 2024-03-29T11:18:05.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.710+0000 7f32abb7b640 1 librados: init done 2024-03-29T11:18:05.698 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.711+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3290001ca0 con 0x7f32a40b4100 2024-03-29T11:18:05.698 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.711+0000 7f32a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:05.698 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.711+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3290003790 con 0x7f32a40b4100 2024-03-29T11:18:05.698 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.711+0000 7f32a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:05.698 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.711+0000 7f32a27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:05.698 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.711+0000 7f32a27fc640 10 monclient: dump: 2024-03-29T11:18:05.698 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:05.698 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:05.698 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:05.699 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:05.699 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:05.699 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:05.699 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:05.699 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:05.699 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:05.699 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:05.699 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.711+0000 7f32a37fe640 4 set_mon_vals no callback set 2024-03-29T11:18:05.699 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.712+0000 7f32abb7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:05.699 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.713+0000 7f32abb7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:05.699 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.713+0000 7f32abb7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:18:05.699 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.713+0000 7f32abb7b640 1 -- 172.21.15.170:0/1916233038 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328c005080 con 0x7f32a40b4100 2024-03-29T11:18:05.699 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.713+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f3290019760 con 0x7f32a40b4100 2024-03-29T11:18:05.700 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.714+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f329004f1e0 con 0x7f32a40b4100 2024-03-29T11:18:05.700 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.714+0000 7f32a27fc640 10 client.4447.objecter ms_dispatch 0x7f32a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:05.700 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.714+0000 7f32a27fc640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:05.700 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.714+0000 7f32a27fc640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:05.700 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.714+0000 7f32a27fc640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-29T11:18:05.700 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.714+0000 7f32a9917640 10 monclient: get_auth_request con 0x7f328806e2d0 auth_method 0 2024-03-29T11:18:05.700 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.714+0000 7f32a27fc640 10 client.4447.objecter ms_handle_connect 0x7f328806e2d0 2024-03-29T11:18:05.702 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.725+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f329004ee10 con 0x7f32a40b4100 2024-03-29T11:18:05.702 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.725+0000 7f32a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:05.702 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:05.725+0000 7f32a27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:06.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.047+0000 7f32abb7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:06.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.047+0000 7f32abb7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:06.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.047+0000 7f32abb7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:18:06.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.047+0000 7f32abb7b640 1 -- 172.21.15.170:0/1916233038 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f328c005330 con 0x7f32a40b4100 2024-03-29T11:18:06.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.048+0000 7f32a27fc640 1 -- 172.21.15.170:0/1916233038 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f32900035f0 con 0x7f32a40b4100 2024-03-29T11:18:06.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.048+0000 7f32a27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:06.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.048+0000 7f32a27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:06.028 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:18:06.028 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:06.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.055+0000 7f32abb7b640 10 librados: watch_flush enter 2024-03-29T11:18:06.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.055+0000 7f32abb7b640 10 librados: watch_flush exit 2024-03-29T11:18:06.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.055+0000 7f32abb7b640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-29T11:18:06.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.055+0000 7f32abb7b640 10 client.4447.objecter successfully canceled tick 2024-03-29T11:18:06.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.056+0000 7f32abb7b640 1 -- 172.21.15.170:0/1916233038 >> v1:172.21.15.170:6816/3298994444 conn(0x7f328806e2d0 legacy=0x7f3288070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:06.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.056+0000 7f32abb7b640 10 monclient: shutdown 2024-03-29T11:18:06.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.056+0000 7f32abb7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:06.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.056+0000 7f32abb7b640 1 -- 172.21.15.170:0/1916233038 >> v1:172.21.15.184:6789/0 conn(0x7f32a40b4100 legacy=0x7f32a41405d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:06.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.057+0000 7f32abb7b640 1 -- 172.21.15.170:0/1916233038 shutdown_connections 2024-03-29T11:18:06.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.057+0000 7f32abb7b640 1 -- 172.21.15.170:0/1916233038 >> 172.21.15.170:0/1916233038 conn(0x7f32a40b35e0 msgr2=0x7f32a413b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:06.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.057+0000 7f32abb7b640 1 -- 172.21.15.170:0/1916233038 shutdown_connections 2024-03-29T11:18:06.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.057+0000 7f32abb7b640 1 -- 172.21.15.170:0/1916233038 wait complete. 2024-03-29T11:18:06.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:06.057+0000 7f32abb7b640 1 librados: shutdown 2024-03-29T11:18:06.052 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:12.860 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:13.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.029+0000 7fa7b3e9a640 10 monclient: get_monmap_and_config 2024-03-29T11:18:13.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.029+0000 7fa7b3e9a640 10 monclient: build_initial_monmap 2024-03-29T11:18:13.007 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.029+0000 7fa7b3e9a640 10 monclient: monmap: 2024-03-29T11:18:13.007 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:18:13.007 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:13.007 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:18:13.031002+0000 2024-03-29T11:18:13.007 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:18:13.031002+0000 2024-03-29T11:18:13.007 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:13.007 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:13.008 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:18:13.008 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:18:13.008 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:18:13.008 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:13.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.030+0000 7fa7b3e9a640 1 Processor -- start 2024-03-29T11:18:13.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.030+0000 7fa7b3e9a640 1 -- start start 2024-03-29T11:18:13.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.030+0000 7fa7b3e9a640 10 monclient: init 2024-03-29T11:18:13.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.031+0000 7fa7b3e9a640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:13.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.031+0000 7fa7b3e9a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:13.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.032+0000 7fa7b3e9a640 10 monclient(hunting): picked mon.noname-a con 0x7fa7ac1457f0 addr v1:172.21.15.170:6789/0 2024-03-29T11:18:13.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.032+0000 7fa7b3e9a640 10 monclient(hunting): picked mon.noname-c con 0x7fa7ac13f650 addr v1:172.21.15.184:6789/0 2024-03-29T11:18:13.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.032+0000 7fa7b3e9a640 10 monclient(hunting): picked mon.noname-b con 0x7fa7ac0a9d70 addr v1:172.21.15.170:6790/0 2024-03-29T11:18:13.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.032+0000 7fa7b3e9a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac0aaef0 con 0x7fa7ac1457f0 2024-03-29T11:18:13.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.032+0000 7fa7b3e9a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac145bd0 con 0x7fa7ac13f650 2024-03-29T11:18:13.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.032+0000 7fa7b3e9a640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac145eb0 con 0x7fa7ac0a9d70 2024-03-29T11:18:13.010 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.032+0000 7fa7b3e9a640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:13.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.032+0000 7fa7b3e9a640 10 monclient(hunting): authenticate will time out at 1274.344440s 2024-03-29T11:18:13.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.037+0000 7fa7b1c36640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fa7ac1457f0 0x7fa7ac13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:39943/0 (socket says 172.21.15.170:39943) 2024-03-29T11:18:13.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.037+0000 7fa7b1c36640 1 -- 172.21.15.170:0/2266714399 learned_addr learned my addr 172.21.15.170:0/2266714399 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:18:13.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.040+0000 7fa7b0c34640 1 -- 172.21.15.170:0/2266714399 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20252787 0 0) 0x7fa7ac145bd0 con 0x7fa7ac13f650 2024-03-29T11:18:13.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.040+0000 7fa7b0c34640 10 monclient(hunting): _negotiate 2024-03-29T11:18:13.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.040+0000 7fa7b0c34640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:13.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.040+0000 7fa7b0c34640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:13.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.040+0000 7fa7b0c34640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:18:13.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.040+0000 7fa7b0c34640 1 -- 172.21.15.170:0/2266714399 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa798003520 con 0x7fa7ac13f650 2024-03-29T11:18:13.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 1 -- 172.21.15.170:0/2266714399 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4063511836 0 0) 0x7fa7ac0aaef0 con 0x7fa7ac1457f0 2024-03-29T11:18:13.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 10 monclient(hunting): _negotiate 2024-03-29T11:18:13.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:13.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:13.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:18:13.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 1 -- 172.21.15.170:0/2266714399 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7ac145bd0 con 0x7fa7ac1457f0 2024-03-29T11:18:13.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 1 -- 172.21.15.170:0/2266714399 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157431510 0 0) 0x7fa7ac145eb0 con 0x7fa7ac0a9d70 2024-03-29T11:18:13.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 10 monclient(hunting): _negotiate 2024-03-29T11:18:13.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:13.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:13.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:18:13.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 1 -- 172.21.15.170:0/2266714399 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7ac0aaef0 con 0x7fa7ac0a9d70 2024-03-29T11:18:13.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 1 -- 172.21.15.170:0/2266714399 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4173945961 0 0) 0x7fa7ac0aaef0 con 0x7fa7ac0a9d70 2024-03-29T11:18:13.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 1 -- 172.21.15.170:0/2266714399 >> v1:172.21.15.184:6789/0 conn(0x7fa7ac13f650 legacy=0x7fa7ac0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:13.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 1 -- 172.21.15.170:0/2266714399 >> v1:172.21.15.170:6789/0 conn(0x7fa7ac1457f0 legacy=0x7fa7ac13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:13.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 10 monclient: _finish_hunting 0 2024-03-29T11:18:13.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 1 monclient: found mon.noname-b 2024-03-29T11:18:13.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:13.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:18:13.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 1 -- 172.21.15.170:0/2266714399 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ac146340 con 0x7fa7ac0a9d70 2024-03-29T11:18:13.021 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 10 monclient: _finish_auth 0 2024-03-29T11:18:13.022 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 10 monclient: _check_auth_tickets 2024-03-29T11:18:13.022 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:13.022 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 1 -- 172.21.15.170:0/2266714399 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa7a40018b0 con 0x7fa7ac0a9d70 2024-03-29T11:18:13.022 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.042+0000 7fa7b0c34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:13.022 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.043+0000 7fa7b0c34640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:13.022 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.043+0000 7fa7b0c34640 10 monclient: dump: 2024-03-29T11:18:13.022 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:13.022 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:13.022 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:13.022 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:13.022 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:13.022 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:13.022 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:13.022 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:13.023 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:13.023 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:13.023 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.043+0000 7fa7b3e9a640 5 monclient: authenticate success, global_id 4487 2024-03-29T11:18:13.023 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.043+0000 7fa7b3e9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:13.023 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.043+0000 7fa7b0c34640 1 -- 172.21.15.170:0/2266714399 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7a4001a10 con 0x7fa7ac0a9d70 2024-03-29T11:18:13.023 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.043+0000 7fa7b0c34640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:13.023 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.043+0000 7fa7b0c34640 1 -- 172.21.15.170:0/2266714399 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa7a4001e20 con 0x7fa7ac0a9d70 2024-03-29T11:18:13.023 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.043+0000 7fa7b0c34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:13.023 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.043+0000 7fa7b0c34640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:13.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.043+0000 7fa7b0c34640 10 monclient: dump: 2024-03-29T11:18:13.024 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:13.024 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:13.024 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:13.024 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:13.024 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:13.024 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:13.024 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:13.024 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:13.024 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:13.024 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:13.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.043+0000 7fa7b3e9a640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:13.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.043+0000 7fa7b3e9a640 4 set_mon_vals no callback set 2024-03-29T11:18:13.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.044+0000 7fa7b3e9a640 10 monclient: shutdown 2024-03-29T11:18:13.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.044+0000 7fa7b3e9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:13.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.044+0000 7fa7b3e9a640 1 -- 172.21.15.170:0/2266714399 >> v1:172.21.15.170:6790/0 conn(0x7fa7ac0a9d70 legacy=0x7fa7ac14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:13.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.046+0000 7fa7b3e9a640 1 -- 172.21.15.170:0/2266714399 shutdown_connections 2024-03-29T11:18:13.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.046+0000 7fa7b3e9a640 1 -- 172.21.15.170:0/2266714399 >> 172.21.15.170:0/2266714399 conn(0x7fa7ac066a00 msgr2=0x7fa7ac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:13.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.047+0000 7fa7b3e9a640 1 -- 172.21.15.170:0/2266714399 shutdown_connections 2024-03-29T11:18:13.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.047+0000 7fa7b3e9a640 1 -- 172.21.15.170:0/2266714399 wait complete. 2024-03-29T11:18:13.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.048+0000 7fa7b3e9a640 10 monclient: build_initial_monmap 2024-03-29T11:18:13.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.048+0000 7fa7b3e9a640 10 monclient: monmap: 2024-03-29T11:18:13.029 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:18:13.029 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:13.029 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:18:13.029 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:18:13.029 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:13.029 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:13.029 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:18:13.030 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:18:13.030 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:18:13.030 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:13.030 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.048+0000 7fa7b3e9a640 1 librados: starting msgr at 2024-03-29T11:18:13.030 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.048+0000 7fa7b3e9a640 1 librados: starting objecter 2024-03-29T11:18:13.030 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.049+0000 7fa7b3e9a640 1 Processor -- start 2024-03-29T11:18:13.030 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.050+0000 7fa7b3e9a640 1 -- start start 2024-03-29T11:18:13.030 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.050+0000 7fa7b3e9a640 1 librados: setting wanted keys 2024-03-29T11:18:13.030 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.050+0000 7fa7b3e9a640 1 librados: calling monclient init 2024-03-29T11:18:13.030 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.050+0000 7fa7b3e9a640 10 monclient: init 2024-03-29T11:18:13.030 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.051+0000 7fa7b3e9a640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:13.030 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.051+0000 7fa7b3e9a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:18:13.030 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.051+0000 7fa7b3e9a640 10 monclient(hunting): picked mon.noname-c con 0x7fa7ac0a9d70 addr v1:172.21.15.170:6790/0 2024-03-29T11:18:13.030 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.051+0000 7fa7b3e9a640 10 monclient(hunting): picked mon.noname-a con 0x7fa7ac13f650 addr v1:172.21.15.170:6789/0 2024-03-29T11:18:13.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.051+0000 7fa7b3e9a640 10 monclient(hunting): picked mon.noname-b con 0x7fa7ac1457f0 addr v1:172.21.15.184:6789/0 2024-03-29T11:18:13.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.051+0000 7fa7b3e9a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac142c40 con 0x7fa7ac13f650 2024-03-29T11:18:13.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.051+0000 7fa7b3e9a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac06f8c0 con 0x7fa7ac1457f0 2024-03-29T11:18:13.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.051+0000 7fa7b3e9a640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac142650 con 0x7fa7ac0a9d70 2024-03-29T11:18:13.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.051+0000 7fa7b3e9a640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:13.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.051+0000 7fa7b3e9a640 10 monclient(hunting): authenticate will time out at 1274.363516s 2024-03-29T11:18:13.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.055+0000 7fa7b1435640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fa7ac13f650 0x7fa7ac1db720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:40263/0 (socket says 172.21.15.170:40263) 2024-03-29T11:18:13.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.056+0000 7fa7b1435640 1 -- 172.21.15.170:0/1055013297 learned_addr learned my addr 172.21.15.170:0/1055013297 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:18:13.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.056+0000 7fa7a27fc640 10 client.?.objecter ms_handle_connect 0x7fa7ac13f650 2024-03-29T11:18:13.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.056+0000 7fa7a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:13.036 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.059+0000 7fa7a27fc640 1 -- 172.21.15.170:0/1055013297 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644197581 0 0) 0x7fa7ac142c40 con 0x7fa7ac13f650 2024-03-29T11:18:13.036 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.059+0000 7fa7a27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:13.037 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.059+0000 7fa7a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:13.037 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.059+0000 7fa7a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:13.037 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.059+0000 7fa7a27fc640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:18:13.037 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.059+0000 7fa7a27fc640 1 -- 172.21.15.170:0/1055013297 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa790004eb0 con 0x7fa7ac13f650 2024-03-29T11:18:13.037 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.059+0000 7fa7a27fc640 10 client.?.objecter ms_handle_connect 0x7fa7ac1457f0 2024-03-29T11:18:13.037 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.059+0000 7fa7a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:13.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.059+0000 7fa7a27fc640 1 -- 172.21.15.170:0/1055013297 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3850548723 0 0) 0x7fa790004eb0 con 0x7fa7ac13f650 2024-03-29T11:18:13.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.059+0000 7fa7a27fc640 1 -- 172.21.15.170:0/1055013297 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa790005300 con 0x7fa7ac13f650 2024-03-29T11:18:13.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.059+0000 7fa7a27fc640 10 client.?.objecter ms_handle_connect 0x7fa7ac0a9d70 2024-03-29T11:18:13.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.059+0000 7fa7a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:13.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.059+0000 7fa7a27fc640 1 -- 172.21.15.170:0/1055013297 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa7a8001a10 con 0x7fa7ac13f650 2024-03-29T11:18:13.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.059+0000 7fa7a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:13.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.059+0000 7fa7a27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:13.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.059+0000 7fa7a27fc640 10 monclient(hunting): dump: 2024-03-29T11:18:13.039 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:13.039 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:13.039 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:13.040 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:13.040 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:13.040 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:13.040 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:13.040 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:13.040 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:13.040 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:13.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.059+0000 7fa7a27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:13.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.060+0000 7fa7a27fc640 1 -- 172.21.15.170:0/1055013297 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2866477656 0 0) 0x7fa790005300 con 0x7fa7ac13f650 2024-03-29T11:18:13.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.060+0000 7fa7a27fc640 1 -- 172.21.15.170:0/1055013297 >> v1:172.21.15.170:6790/0 conn(0x7fa7ac0a9d70 legacy=0x7fa7ac06e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:13.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.060+0000 7fa7a27fc640 1 -- 172.21.15.170:0/1055013297 >> v1:172.21.15.184:6789/0 conn(0x7fa7ac1457f0 legacy=0x7fa7ac1dbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:13.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.060+0000 7fa7a27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:13.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.060+0000 7fa7a27fc640 1 monclient: found mon.a 2024-03-29T11:18:13.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.060+0000 7fa7a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:13.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.060+0000 7fa7a27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:13.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.060+0000 7fa7a27fc640 1 -- 172.21.15.170:0/1055013297 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ac1427e0 con 0x7fa7ac13f650 2024-03-29T11:18:13.041 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.060+0000 7fa7a27fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:13.041 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.060+0000 7fa7a27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:13.041 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.060+0000 7fa7a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:13.041 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.061+0000 7fa7a27fc640 1 -- 172.21.15.170:0/1055013297 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7a8001cd0 con 0x7fa7ac13f650 2024-03-29T11:18:13.041 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.061+0000 7fa7a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:13.041 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.061+0000 7fa7a3fff640 4 set_mon_vals no callback set 2024-03-29T11:18:13.042 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.064+0000 7fa7b3e9a640 5 monclient: authenticate success, global_id 4455 2024-03-29T11:18:13.042 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.064+0000 7fa7b3e9a640 10 monclient: _renew_subs 2024-03-29T11:18:13.042 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.064+0000 7fa7b3e9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:13.042 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.064+0000 7fa7b3e9a640 1 -- 172.21.15.170:0/1055013297 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7ac06e810 con 0x7fa7ac13f650 2024-03-29T11:18:13.042 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.064+0000 7fa7b3e9a640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:13.042 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.064+0000 7fa7b3e9a640 10 monclient: _renew_subs 2024-03-29T11:18:13.042 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.064+0000 7fa7b3e9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:13.042 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.064+0000 7fa7b3e9a640 1 -- 172.21.15.170:0/1055013297 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7ac1e7250 con 0x7fa7ac13f650 2024-03-29T11:18:13.042 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.064+0000 7fa7b3e9a640 1 librados: init done 2024-03-29T11:18:13.043 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.065+0000 7fa7a27fc640 1 -- 172.21.15.170:0/1055013297 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa7a8001a10 con 0x7fa7ac13f650 2024-03-29T11:18:13.044 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.065+0000 7fa7a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:13.044 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.065+0000 7fa7a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:13.044 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.065+0000 7fa7a27fc640 10 monclient: dump: 2024-03-29T11:18:13.044 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:13.044 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:13.044 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:13.044 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:13.044 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:13.044 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:13.044 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:13.044 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:13.044 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:13.044 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:13.044 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.066+0000 7fa7b3e9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:13.045 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.066+0000 7fa7b3e9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:13.045 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.066+0000 7fa7b3e9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:13.045 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.066+0000 7fa7b3e9a640 1 -- 172.21.15.170:0/1055013297 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7ac142a10 con 0x7fa7ac13f650 2024-03-29T11:18:13.045 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.067+0000 7fa7a27fc640 1 -- 172.21.15.170:0/1055013297 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fa7a8003940 con 0x7fa7ac13f650 2024-03-29T11:18:13.046 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.067+0000 7fa7a27fc640 1 -- 172.21.15.170:0/1055013297 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fa7a80848d0 con 0x7fa7ac13f650 2024-03-29T11:18:13.046 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.067+0000 7fa7a27fc640 10 client.4455.objecter ms_dispatch 0x7fa7ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:13.046 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.067+0000 7fa7a27fc640 3 client.4455.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:13.046 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.067+0000 7fa7a27fc640 3 client.4455.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:13.046 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.067+0000 7fa7a27fc640 20 client.4455.objecter dump_active .. 0 homeless 2024-03-29T11:18:13.047 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.069+0000 7fa7b1c36640 10 monclient: get_auth_request con 0x7fa79006e0c0 auth_method 0 2024-03-29T11:18:13.047 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.070+0000 7fa7a27fc640 10 client.4455.objecter ms_handle_connect 0x7fa79006e0c0 2024-03-29T11:18:13.053 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.076+0000 7fa7a27fc640 1 -- 172.21.15.170:0/1055013297 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7ac142a10 con 0x7fa7ac13f650 2024-03-29T11:18:13.053 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.076+0000 7fa7a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:13.054 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.076+0000 7fa7a27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:13.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.398+0000 7fa7b3e9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:13.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.398+0000 7fa7b3e9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:13.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.398+0000 7fa7b3e9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:13.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.398+0000 7fa7b3e9a640 1 -- 172.21.15.170:0/1055013297 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7ac142a10 con 0x7fa7ac13f650 2024-03-29T11:18:13.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.399+0000 7fa7a27fc640 1 -- 172.21.15.170:0/1055013297 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fa7ac142a10 con 0x7fa7ac13f650 2024-03-29T11:18:13.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.399+0000 7fa7a27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:13.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.399+0000 7fa7a27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:13.377 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:18:13.377 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:13.381 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.403+0000 7fa7b3e9a640 10 librados: watch_flush enter 2024-03-29T11:18:13.381 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.404+0000 7fa7b3e9a640 10 librados: watch_flush exit 2024-03-29T11:18:13.381 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.404+0000 7fa7b3e9a640 20 client.4455.objecter shutdown clearing up homeless session... 2024-03-29T11:18:13.382 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.404+0000 7fa7b3e9a640 10 client.4455.objecter successfully canceled tick 2024-03-29T11:18:13.382 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.404+0000 7fa7b3e9a640 1 -- 172.21.15.170:0/1055013297 >> v1:172.21.15.170:6816/3298994444 conn(0x7fa79006e0c0 legacy=0x7fa790070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:13.382 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.404+0000 7fa7b3e9a640 10 monclient: shutdown 2024-03-29T11:18:13.382 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.404+0000 7fa7b3e9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:13.382 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.404+0000 7fa7b3e9a640 1 -- 172.21.15.170:0/1055013297 >> v1:172.21.15.170:6789/0 conn(0x7fa7ac13f650 legacy=0x7fa7ac1db720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:13.382 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.404+0000 7fa7b3e9a640 1 -- 172.21.15.170:0/1055013297 shutdown_connections 2024-03-29T11:18:13.382 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.404+0000 7fa7b3e9a640 1 -- 172.21.15.170:0/1055013297 >> 172.21.15.170:0/1055013297 conn(0x7fa7ac066a00 msgr2=0x7fa7ac13e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:13.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.406+0000 7fa7b3e9a640 1 -- 172.21.15.170:0/1055013297 shutdown_connections 2024-03-29T11:18:13.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.407+0000 7fa7b3e9a640 1 -- 172.21.15.170:0/1055013297 wait complete. 2024-03-29T11:18:13.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:13.407+0000 7fa7b3e9a640 1 librados: shutdown 2024-03-29T11:18:13.402 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:20.209 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:20.365 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.389+0000 7f3755a93640 10 monclient: get_monmap_and_config 2024-03-29T11:18:20.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.389+0000 7f3755a93640 10 monclient: build_initial_monmap 2024-03-29T11:18:20.366 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.389+0000 7f3755a93640 10 monclient: monmap: 2024-03-29T11:18:20.366 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:18:20.366 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:20.366 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:18:20.389687+0000 2024-03-29T11:18:20.366 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:18:20.389687+0000 2024-03-29T11:18:20.366 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:20.366 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:20.366 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:18:20.366 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:18:20.366 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:18:20.366 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:20.367 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.390+0000 7f3755a93640 1 Processor -- start 2024-03-29T11:18:20.367 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.390+0000 7f3755a93640 1 -- start start 2024-03-29T11:18:20.367 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.390+0000 7f3755a93640 10 monclient: init 2024-03-29T11:18:20.367 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.391+0000 7f3755a93640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:20.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.391+0000 7f3755a93640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:20.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.391+0000 7f3755a93640 10 monclient(hunting): picked mon.noname-c con 0x7f37500b1f60 addr v1:172.21.15.184:6789/0 2024-03-29T11:18:20.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.391+0000 7f3755a93640 10 monclient(hunting): picked mon.noname-b con 0x7f3750071eb0 addr v1:172.21.15.170:6790/0 2024-03-29T11:18:20.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.391+0000 7f3755a93640 10 monclient(hunting): picked mon.noname-a con 0x7f3750143d10 addr v1:172.21.15.170:6789/0 2024-03-29T11:18:20.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.391+0000 7f3755a93640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375006ef80 con 0x7f3750143d10 2024-03-29T11:18:20.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.391+0000 7f3755a93640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375006f260 con 0x7f37500b1f60 2024-03-29T11:18:20.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.391+0000 7f3755a93640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37500b40c0 con 0x7f3750071eb0 2024-03-29T11:18:20.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.391+0000 7f3755a93640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:20.375 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.391+0000 7f3755a93640 10 monclient(hunting): authenticate will time out at 1281.702289s 2024-03-29T11:18:20.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.394+0000 7f374ffff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f3750143d10 0x7f3750146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:58674/0 (socket says 172.21.15.170:58674) 2024-03-29T11:18:20.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.394+0000 7f374ffff640 1 -- 172.21.15.170:0/1306624018 learned_addr learned my addr 172.21.15.170:0/1306624018 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:18:20.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.394+0000 7f374e7fc640 1 -- 172.21.15.170:0/1306624018 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2805264857 0 0) 0x7f375006ef80 con 0x7f3750143d10 2024-03-29T11:18:20.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.394+0000 7f374e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:20.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.394+0000 7f374e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:20.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.394+0000 7f374e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:20.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.394+0000 7f374e7fc640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:18:20.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.394+0000 7f374e7fc640 1 -- 172.21.15.170:0/1306624018 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f373c003520 con 0x7f3750143d10 2024-03-29T11:18:20.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 1 -- 172.21.15.170:0/1306624018 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439117562 0 0) 0x7f37500b40c0 con 0x7f3750071eb0 2024-03-29T11:18:20.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:20.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:20.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:20.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:18:20.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 1 -- 172.21.15.170:0/1306624018 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f375006ef80 con 0x7f3750071eb0 2024-03-29T11:18:20.376 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 1 -- 172.21.15.170:0/1306624018 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1868180590 0 0) 0x7f373c003520 con 0x7f3750143d10 2024-03-29T11:18:20.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 1 -- 172.21.15.170:0/1306624018 >> v1:172.21.15.170:6790/0 conn(0x7f3750071eb0 legacy=0x7f3750146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:20.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 1 -- 172.21.15.170:0/1306624018 >> v1:172.21.15.184:6789/0 conn(0x7f37500b1f60 legacy=0x7f3750143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:20.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:20.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 1 monclient: found mon.noname-a 2024-03-29T11:18:20.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:20.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:18:20.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 1 -- 172.21.15.170:0/1306624018 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3750155d70 con 0x7f3750143d10 2024-03-29T11:18:20.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:20.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:20.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:20.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 1 -- 172.21.15.170:0/1306624018 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3748001a10 con 0x7f3750143d10 2024-03-29T11:18:20.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:20.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:20.377 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 10 monclient: dump: 2024-03-29T11:18:20.377 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:20.377 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:20.377 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:20.378 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:20.378 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:20.378 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:20.378 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:20.378 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:20.378 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:20.378 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:20.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f3755a93640 5 monclient: authenticate success, global_id 4458 2024-03-29T11:18:20.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f3755a93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:20.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 1 -- 172.21.15.170:0/1306624018 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3748001b70 con 0x7f3750143d10 2024-03-29T11:18:20.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:20.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 1 -- 172.21.15.170:0/1306624018 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3748001f80 con 0x7f3750143d10 2024-03-29T11:18:20.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:20.378 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:20.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f374e7fc640 10 monclient: dump: 2024-03-29T11:18:20.379 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:20.379 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:20.379 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:20.379 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:20.379 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:20.379 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:20.379 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:20.379 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:20.379 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:20.379 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:20.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f3755a93640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:20.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.395+0000 7f3755a93640 4 set_mon_vals no callback set 2024-03-29T11:18:20.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.397+0000 7f3755a93640 10 monclient: shutdown 2024-03-29T11:18:20.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.397+0000 7f3755a93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:20.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.397+0000 7f3755a93640 1 -- 172.21.15.170:0/1306624018 >> v1:172.21.15.170:6789/0 conn(0x7f3750143d10 legacy=0x7f3750146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:20.379 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.397+0000 7f3755a93640 1 -- 172.21.15.170:0/1306624018 shutdown_connections 2024-03-29T11:18:20.380 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.397+0000 7f3755a93640 1 -- 172.21.15.170:0/1306624018 >> 172.21.15.170:0/1306624018 conn(0x7f3750066a00 msgr2=0x7f3750066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:20.380 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.397+0000 7f3755a93640 1 -- 172.21.15.170:0/1306624018 shutdown_connections 2024-03-29T11:18:20.380 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.397+0000 7f3755a93640 1 -- 172.21.15.170:0/1306624018 wait complete. 2024-03-29T11:18:20.380 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.398+0000 7f3755a93640 10 monclient: build_initial_monmap 2024-03-29T11:18:20.380 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.398+0000 7f3755a93640 10 monclient: monmap: 2024-03-29T11:18:20.380 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:18:20.380 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:20.380 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:18:20.380 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:18:20.380 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:20.380 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:20.380 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:18:20.380 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:18:20.380 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:18:20.380 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:20.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.399+0000 7f3755a93640 1 librados: starting msgr at 2024-03-29T11:18:20.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.399+0000 7f3755a93640 1 librados: starting objecter 2024-03-29T11:18:20.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.399+0000 7f3755a93640 1 Processor -- start 2024-03-29T11:18:20.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.399+0000 7f3755a93640 1 -- start start 2024-03-29T11:18:20.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.399+0000 7f3755a93640 1 librados: setting wanted keys 2024-03-29T11:18:20.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.399+0000 7f3755a93640 1 librados: calling monclient init 2024-03-29T11:18:20.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.399+0000 7f3755a93640 10 monclient: init 2024-03-29T11:18:20.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.400+0000 7f3755a93640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:20.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.400+0000 7f3755a93640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:20.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.400+0000 7f3755a93640 10 monclient(hunting): picked mon.noname-c con 0x7f37500b1f60 addr v1:172.21.15.170:6790/0 2024-03-29T11:18:20.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.400+0000 7f3755a93640 10 monclient(hunting): picked mon.noname-b con 0x7f3750143d10 addr v1:172.21.15.184:6789/0 2024-03-29T11:18:20.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.400+0000 7f3755a93640 10 monclient(hunting): picked mon.noname-a con 0x7f3750154f70 addr v1:172.21.15.170:6789/0 2024-03-29T11:18:20.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.400+0000 7f3755a93640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375014b620 con 0x7f3750154f70 2024-03-29T11:18:20.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.400+0000 7f3755a93640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375013fdd0 con 0x7f3750143d10 2024-03-29T11:18:20.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.400+0000 7f3755a93640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3750149ae0 con 0x7f37500b1f60 2024-03-29T11:18:20.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.400+0000 7f3755a93640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:20.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.400+0000 7f3755a93640 10 monclient(hunting): authenticate will time out at 1281.711371s 2024-03-29T11:18:20.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.401+0000 7f374ffff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f3750154f70 0x7f37501e6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:34312/0 (socket says 172.21.15.170:34312) 2024-03-29T11:18:20.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.401+0000 7f374ffff640 1 -- 172.21.15.170:0/661878083 learned_addr learned my addr 172.21.15.170:0/661878083 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:18:20.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.404+0000 7f3747fff640 10 client.?.objecter ms_handle_connect 0x7f3750154f70 2024-03-29T11:18:20.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.404+0000 7f3747fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:20.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.404+0000 7f3747fff640 10 client.?.objecter ms_handle_connect 0x7f3750143d10 2024-03-29T11:18:20.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.404+0000 7f3747fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:20.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.404+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 108021820 0 0) 0x7f375013fdd0 con 0x7f3750143d10 2024-03-29T11:18:20.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.404+0000 7f3747fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:20.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.404+0000 7f3747fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:20.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.404+0000 7f3747fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:20.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.404+0000 7f3747fff640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:18:20.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.404+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3730004eb0 con 0x7f3750143d10 2024-03-29T11:18:20.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.405+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395858156 0 0) 0x7f375014b620 con 0x7f3750154f70 2024-03-29T11:18:20.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.405+0000 7f3747fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:20.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.405+0000 7f3747fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:20.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.405+0000 7f3747fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:20.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.405+0000 7f3747fff640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:18:20.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.405+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f375013fdd0 con 0x7f3750154f70 2024-03-29T11:18:20.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.405+0000 7f3747fff640 10 client.?.objecter ms_handle_connect 0x7f37500b1f60 2024-03-29T11:18:20.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.405+0000 7f3747fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:20.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.405+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1130843134 0 0) 0x7f375013fdd0 con 0x7f3750154f70 2024-03-29T11:18:20.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.405+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f375014b620 con 0x7f3750154f70 2024-03-29T11:18:20.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1708077893 0 0) 0x7f3730004eb0 con 0x7f3750143d10 2024-03-29T11:18:20.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f375013fdd0 con 0x7f3750143d10 2024-03-29T11:18:20.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3748001e80 con 0x7f3750154f70 2024-03-29T11:18:20.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:20.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:20.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 10 monclient(hunting): dump: 2024-03-29T11:18:20.386 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:20.387 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:20.387 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:20.387 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:20.387 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:20.387 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:20.387 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:20.387 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:20.387 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:20.387 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:20.387 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:20.387 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3740001910 con 0x7f3750143d10 2024-03-29T11:18:20.387 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:20.387 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:20.387 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 10 monclient(hunting): dump: 2024-03-29T11:18:20.387 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:20.387 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:20.387 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:20.388 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:20.388 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:20.388 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:20.388 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:20.388 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:20.388 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:20.388 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:20.388 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4048780806 0 0) 0x7f375014b620 con 0x7f3750154f70 2024-03-29T11:18:20.388 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 >> v1:172.21.15.170:6790/0 conn(0x7f37500b1f60 legacy=0x7f375013f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:20.388 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 >> v1:172.21.15.184:6789/0 conn(0x7f3750143d10 legacy=0x7f3750148f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:20.388 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:20.388 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 1 monclient: found mon.a 2024-03-29T11:18:20.388 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:20.388 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:20.389 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3750149640 con 0x7f3750154f70 2024-03-29T11:18:20.389 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 10 monclient: _finish_auth 0 2024-03-29T11:18:20.389 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:20.389 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3747fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:20.389 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3755a93640 5 monclient: authenticate success, global_id 4461 2024-03-29T11:18:20.389 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3755a93640 10 monclient: _renew_subs 2024-03-29T11:18:20.389 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3755a93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:20.389 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3755a93640 1 -- 172.21.15.170:0/661878083 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3750149fc0 con 0x7f3750154f70 2024-03-29T11:18:20.389 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3755a93640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:20.389 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3755a93640 10 monclient: _renew_subs 2024-03-29T11:18:20.389 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3755a93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:20.389 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3755a93640 1 -- 172.21.15.170:0/661878083 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37501e7840 con 0x7f3750154f70 2024-03-29T11:18:20.389 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.406+0000 7f3755a93640 1 librados: init done 2024-03-29T11:18:20.391 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.407+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37480015b0 con 0x7f3750154f70 2024-03-29T11:18:20.392 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.407+0000 7f3747fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:20.392 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.407+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3748001fe0 con 0x7f3750154f70 2024-03-29T11:18:20.392 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.407+0000 7f3747fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:20.392 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.407+0000 7f3747fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:20.392 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.407+0000 7f3747fff640 10 monclient: dump: 2024-03-29T11:18:20.392 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:20.392 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:20.392 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:20.392 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:20.392 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:20.392 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:20.392 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:20.392 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:20.392 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:20.393 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:20.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.407+0000 7f374d7fa640 4 set_mon_vals no callback set 2024-03-29T11:18:20.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.409+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f374801ae20 con 0x7f3750154f70 2024-03-29T11:18:20.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.410+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f3748084d60 con 0x7f3750154f70 2024-03-29T11:18:20.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.410+0000 7f3747fff640 10 client.4461.objecter ms_dispatch 0x7f3750000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:20.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.410+0000 7f3747fff640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:20.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.410+0000 7f3747fff640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:20.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.410+0000 7f3747fff640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-29T11:18:20.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.410+0000 7f374f7fe640 10 monclient: get_auth_request con 0x7f373006df30 auth_method 0 2024-03-29T11:18:20.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.410+0000 7f3747fff640 10 client.4461.objecter ms_handle_connect 0x7f373006df30 2024-03-29T11:18:20.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.411+0000 7f3755a93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:20.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.411+0000 7f3755a93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:20.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.411+0000 7f3755a93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:20.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.411+0000 7f3755a93640 1 -- 172.21.15.170:0/661878083 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3734005080 con 0x7f3750154f70 2024-03-29T11:18:20.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.422+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3748050470 con 0x7f3750154f70 2024-03-29T11:18:20.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.422+0000 7f3747fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:20.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.422+0000 7f3747fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:20.723 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.747+0000 7f3755a93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:20.724 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.747+0000 7f3755a93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:20.724 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.747+0000 7f3755a93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:20.724 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.747+0000 7f3755a93640 1 -- 172.21.15.170:0/661878083 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3734005330 con 0x7f3750154f70 2024-03-29T11:18:20.724 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.748+0000 7f3747fff640 1 -- 172.21.15.170:0/661878083 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f3748052570 con 0x7f3750154f70 2024-03-29T11:18:20.724 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.748+0000 7f3747fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:20.724 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.748+0000 7f3747fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:20.728 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:18:20.728 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:20.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.751+0000 7f3755a93640 10 librados: watch_flush enter 2024-03-29T11:18:20.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.751+0000 7f3755a93640 10 librados: watch_flush exit 2024-03-29T11:18:20.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.751+0000 7f3755a93640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-29T11:18:20.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.751+0000 7f3755a93640 10 client.4461.objecter successfully canceled tick 2024-03-29T11:18:20.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.751+0000 7f3755a93640 1 -- 172.21.15.170:0/661878083 >> v1:172.21.15.170:6816/3298994444 conn(0x7f373006df30 legacy=0x7f37300703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:20.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.751+0000 7f3755a93640 10 monclient: shutdown 2024-03-29T11:18:20.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.751+0000 7f3755a93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:20.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.751+0000 7f3755a93640 1 -- 172.21.15.170:0/661878083 >> v1:172.21.15.170:6789/0 conn(0x7f3750154f70 legacy=0x7f37501e6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:20.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.752+0000 7f3755a93640 1 -- 172.21.15.170:0/661878083 shutdown_connections 2024-03-29T11:18:20.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.752+0000 7f3755a93640 1 -- 172.21.15.170:0/661878083 >> 172.21.15.170:0/661878083 conn(0x7f3750066a00 msgr2=0x7f37500b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:20.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.752+0000 7f3755a93640 1 -- 172.21.15.170:0/661878083 shutdown_connections 2024-03-29T11:18:20.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.752+0000 7f3755a93640 1 -- 172.21.15.170:0/661878083 wait complete. 2024-03-29T11:18:20.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:20.752+0000 7f3755a93640 1 librados: shutdown 2024-03-29T11:18:20.750 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:27.557 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:27.715 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.731+0000 7f4a8424e640 10 monclient: get_monmap_and_config 2024-03-29T11:18:27.715 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.731+0000 7f4a8424e640 10 monclient: build_initial_monmap 2024-03-29T11:18:27.715 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.731+0000 7f4a8424e640 10 monclient: monmap: 2024-03-29T11:18:27.715 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:18:27.715 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:27.715 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:18:27.731554+0000 2024-03-29T11:18:27.716 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:18:27.731554+0000 2024-03-29T11:18:27.716 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:27.716 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:27.716 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:18:27.716 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:18:27.716 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:18:27.716 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:27.716 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.731+0000 7f4a8424e640 1 Processor -- start 2024-03-29T11:18:27.716 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.732+0000 7f4a8424e640 1 -- start start 2024-03-29T11:18:27.716 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.732+0000 7f4a8424e640 10 monclient: init 2024-03-29T11:18:27.716 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.733+0000 7f4a8424e640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:27.716 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.733+0000 7f4a8424e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:27.716 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.733+0000 7f4a8424e640 10 monclient(hunting): picked mon.noname-a con 0x7f4a7c0b0770 addr v1:172.21.15.170:6789/0 2024-03-29T11:18:27.716 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.733+0000 7f4a8424e640 10 monclient(hunting): picked mon.noname-b con 0x7f4a7c14a700 addr v1:172.21.15.170:6790/0 2024-03-29T11:18:27.716 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.733+0000 7f4a8424e640 10 monclient(hunting): picked mon.noname-c con 0x7f4a7c0ac5a0 addr v1:172.21.15.184:6789/0 2024-03-29T11:18:27.717 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.733+0000 7f4a8424e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a7c0aba70 con 0x7f4a7c0b0770 2024-03-29T11:18:27.717 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.733+0000 7f4a8424e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a7c0aecb0 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.717 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.733+0000 7f4a8424e640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a7c0af150 con 0x7f4a7c14a700 2024-03-29T11:18:27.717 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.733+0000 7f4a8424e640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:27.717 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.733+0000 7f4a8424e640 10 monclient(hunting): authenticate will time out at 1289.044898s 2024-03-29T11:18:27.717 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.734+0000 7f4a81fea640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f4a7c0b0770 0x7f4a7c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:50602/0 (socket says 172.21.15.170:50602) 2024-03-29T11:18:27.717 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.734+0000 7f4a81fea640 1 -- 172.21.15.170:0/3995788011 learned_addr learned my addr 172.21.15.170:0/3995788011 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:18:27.717 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.734+0000 7f4a80fe8640 1 -- 172.21.15.170:0/3995788011 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383138781 0 0) 0x7f4a7c0aecb0 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.717 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.734+0000 7f4a80fe8640 10 monclient(hunting): _negotiate 2024-03-29T11:18:27.717 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.735+0000 7f4a80fe8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:27.717 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.735+0000 7f4a80fe8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:27.717 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.735+0000 7f4a80fe8640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:18:27.717 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.735+0000 7f4a80fe8640 1 -- 172.21.15.170:0/3995788011 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a68003520 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.717 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.735+0000 7f4a80fe8640 1 -- 172.21.15.170:0/3995788011 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230407669 0 0) 0x7f4a7c0af150 con 0x7f4a7c14a700 2024-03-29T11:18:27.717 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.735+0000 7f4a80fe8640 10 monclient(hunting): _negotiate 2024-03-29T11:18:27.718 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.735+0000 7f4a80fe8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:27.718 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.735+0000 7f4a80fe8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:27.718 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.735+0000 7f4a80fe8640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:18:27.718 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.735+0000 7f4a80fe8640 1 -- 172.21.15.170:0/3995788011 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a7c0aecb0 con 0x7f4a7c14a700 2024-03-29T11:18:27.718 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.735+0000 7f4a80fe8640 1 -- 172.21.15.170:0/3995788011 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169744595 0 0) 0x7f4a7c0aba70 con 0x7f4a7c0b0770 2024-03-29T11:18:27.718 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.735+0000 7f4a80fe8640 10 monclient(hunting): _negotiate 2024-03-29T11:18:27.718 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.735+0000 7f4a80fe8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:27.718 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.735+0000 7f4a80fe8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:27.718 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.735+0000 7f4a80fe8640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:18:27.718 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.735+0000 7f4a80fe8640 1 -- 172.21.15.170:0/3995788011 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a7c0af150 con 0x7f4a7c0b0770 2024-03-29T11:18:27.718 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.735+0000 7f4a80fe8640 1 -- 172.21.15.170:0/3995788011 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3643880211 0 0) 0x7f4a68003520 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.718 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 1 -- 172.21.15.170:0/3995788011 >> v1:172.21.15.170:6790/0 conn(0x7f4a7c14a700 legacy=0x7f4a7c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:27.718 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 1 -- 172.21.15.170:0/3995788011 >> v1:172.21.15.170:6789/0 conn(0x7f4a7c0b0770 legacy=0x7f4a7c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:27.718 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 10 monclient: _finish_hunting 0 2024-03-29T11:18:27.719 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 1 monclient: found mon.noname-c 2024-03-29T11:18:27.719 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:27.719 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:18:27.719 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 1 -- 172.21.15.170:0/3995788011 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7c0ac9d0 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.719 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 10 monclient: _finish_auth 0 2024-03-29T11:18:27.719 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 10 monclient: _check_auth_tickets 2024-03-29T11:18:27.719 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:27.719 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a8424e640 5 monclient: authenticate success, global_id 4459 2024-03-29T11:18:27.719 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a8424e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:27.719 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 1 -- 172.21.15.170:0/3995788011 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f4a740019f0 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.719 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:27.719 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:27.719 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 10 monclient: dump: 2024-03-29T11:18:27.719 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:27.720 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:27.720 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:27.720 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:27.720 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:27.720 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:27.720 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:27.720 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:27.720 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:27.720 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:27.720 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a8424e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:27.720 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 1 -- 172.21.15.170:0/3995788011 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a74001c80 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.720 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:27.720 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a8424e640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:27.720 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a8424e640 4 set_mon_vals no callback set 2024-03-29T11:18:27.720 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 1 -- 172.21.15.170:0/3995788011 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f4a74002010 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.721 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:27.721 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:27.721 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.736+0000 7f4a80fe8640 10 monclient: dump: 2024-03-29T11:18:27.721 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:27.721 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:27.721 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:27.721 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:27.721 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:27.721 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:27.721 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:27.721 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:27.721 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:27.721 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:27.721 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.737+0000 7f4a8424e640 10 monclient: shutdown 2024-03-29T11:18:27.722 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.737+0000 7f4a8424e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:27.722 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.737+0000 7f4a8424e640 1 -- 172.21.15.170:0/3995788011 >> v1:172.21.15.184:6789/0 conn(0x7f4a7c0ac5a0 legacy=0x7f4a7c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:27.722 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.737+0000 7f4a8424e640 1 -- 172.21.15.170:0/3995788011 shutdown_connections 2024-03-29T11:18:27.722 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.737+0000 7f4a8424e640 1 -- 172.21.15.170:0/3995788011 >> 172.21.15.170:0/3995788011 conn(0x7f4a7c06b660 msgr2=0x7f4a7c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:27.722 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.737+0000 7f4a8424e640 1 -- 172.21.15.170:0/3995788011 shutdown_connections 2024-03-29T11:18:27.722 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.737+0000 7f4a8424e640 1 -- 172.21.15.170:0/3995788011 wait complete. 2024-03-29T11:18:27.722 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.738+0000 7f4a8424e640 10 monclient: build_initial_monmap 2024-03-29T11:18:27.722 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.738+0000 7f4a8424e640 10 monclient: monmap: 2024-03-29T11:18:27.722 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:18:27.722 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:27.722 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:18:27.722 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:18:27.722 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:27.723 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:27.723 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:18:27.723 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:18:27.723 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:18:27.723 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:27.723 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.738+0000 7f4a8424e640 1 librados: starting msgr at 2024-03-29T11:18:27.723 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.738+0000 7f4a8424e640 1 librados: starting objecter 2024-03-29T11:18:27.723 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.739+0000 7f4a8424e640 1 Processor -- start 2024-03-29T11:18:27.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.739+0000 7f4a8424e640 1 -- start start 2024-03-29T11:18:27.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.739+0000 7f4a8424e640 1 librados: setting wanted keys 2024-03-29T11:18:27.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.739+0000 7f4a8424e640 1 librados: calling monclient init 2024-03-29T11:18:27.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.739+0000 7f4a8424e640 10 monclient: init 2024-03-29T11:18:27.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a8424e640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:27.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a8424e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:27.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a8424e640 10 monclient(hunting): picked mon.noname-a con 0x7f4a7c0ac5a0 addr v1:172.21.15.170:6789/0 2024-03-29T11:18:27.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a8424e640 10 monclient(hunting): picked mon.noname-c con 0x7f4a7c0b0770 addr v1:172.21.15.170:6790/0 2024-03-29T11:18:27.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a8424e640 10 monclient(hunting): picked mon.noname-b con 0x7f4a7c14a700 addr v1:172.21.15.184:6789/0 2024-03-29T11:18:27.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a8424e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a7c128360 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a8424e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a7c1285d0 con 0x7f4a7c14a700 2024-03-29T11:18:27.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a8424e640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a7c126330 con 0x7f4a7c0b0770 2024-03-29T11:18:27.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a8424e640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:27.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a8424e640 10 monclient(hunting): authenticate will time out at 1289.051285s 2024-03-29T11:18:27.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a817e9640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f4a7c0b0770 0x7f4a7c126850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:52556/0 (socket says 172.21.15.170:52556) 2024-03-29T11:18:27.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a817e9640 1 -- 172.21.15.170:0/2593924051 learned_addr learned my addr 172.21.15.170:0/2593924051 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:18:27.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a727fc640 10 client.?.objecter ms_handle_connect 0x7f4a7c0b0770 2024-03-29T11:18:27.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:27.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a727fc640 10 client.?.objecter ms_handle_connect 0x7f4a7c0ac5a0 2024-03-29T11:18:27.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:27.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a727fc640 1 -- 172.21.15.170:0/2593924051 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363723452 0 0) 0x7f4a7c128360 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:27.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:27.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:27.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a727fc640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:18:27.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a727fc640 1 -- 172.21.15.170:0/2593924051 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a60004eb0 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a727fc640 1 -- 172.21.15.170:0/2593924051 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141814348 0 0) 0x7f4a7c126330 con 0x7f4a7c0b0770 2024-03-29T11:18:27.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:27.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:27.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:27.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a727fc640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:18:27.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.740+0000 7f4a727fc640 1 -- 172.21.15.170:0/2593924051 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a7c128360 con 0x7f4a7c0b0770 2024-03-29T11:18:27.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.741+0000 7f4a727fc640 1 -- 172.21.15.170:0/2593924051 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3530464022 0 0) 0x7f4a60004eb0 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.741+0000 7f4a727fc640 1 -- 172.21.15.170:0/2593924051 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a7c126330 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.741+0000 7f4a727fc640 1 -- 172.21.15.170:0/2593924051 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f4a78001870 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.741+0000 7f4a727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:27.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.741+0000 7f4a727fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:27.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.741+0000 7f4a727fc640 10 monclient(hunting): dump: 2024-03-29T11:18:27.732 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:27.732 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:27.732 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:27.732 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:27.732 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:27.732 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:27.732 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:27.732 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:27.732 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:27.733 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:27.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.741+0000 7f4a727fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:27.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.741+0000 7f4a727fc640 1 -- 172.21.15.170:0/2593924051 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2015574402 0 0) 0x7f4a7c128360 con 0x7f4a7c0b0770 2024-03-29T11:18:27.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.741+0000 7f4a727fc640 1 -- 172.21.15.170:0/2593924051 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a60004eb0 con 0x7f4a7c0b0770 2024-03-29T11:18:27.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.741+0000 7f4a727fc640 1 -- 172.21.15.170:0/2593924051 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2166421147 0 0) 0x7f4a7c126330 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.741+0000 7f4a727fc640 1 -- 172.21.15.170:0/2593924051 >> v1:172.21.15.170:6790/0 conn(0x7f4a7c0b0770 legacy=0x7f4a7c126850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:27.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.741+0000 7f4a727fc640 1 -- 172.21.15.170:0/2593924051 >> v1:172.21.15.184:6789/0 conn(0x7f4a7c14a700 legacy=0x7f4a7c126f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:27.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.741+0000 7f4a727fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:27.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.741+0000 7f4a727fc640 1 monclient: found mon.a 2024-03-29T11:18:27.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:27.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a727fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:27.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a727fc640 1 -- 172.21.15.170:0/2593924051 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7c125f70 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a727fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:27.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a727fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:27.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:27.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a8424e640 5 monclient: authenticate success, global_id 4467 2024-03-29T11:18:27.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a727fc640 1 -- 172.21.15.170:0/2593924051 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a78001d30 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a8424e640 10 monclient: _renew_subs 2024-03-29T11:18:27.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a8424e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:27.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a8424e640 1 -- 172.21.15.170:0/2593924051 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a7c1dfaa0 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a8424e640 10 client.4467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:27.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a8424e640 10 monclient: _renew_subs 2024-03-29T11:18:27.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a8424e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:27.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a8424e640 1 -- 172.21.15.170:0/2593924051 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a7c1df240 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a8424e640 1 librados: init done 2024-03-29T11:18:27.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:27.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a727fc640 1 -- 172.21.15.170:0/2593924051 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f4a78004980 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:27.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:27.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a727fc640 10 monclient: dump: 2024-03-29T11:18:27.735 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:27.735 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:27.735 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:27.736 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:27.736 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:27.736 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:27.736 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:27.736 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:27.736 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:27.736 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:27.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.742+0000 7f4a73fff640 4 set_mon_vals no callback set 2024-03-29T11:18:27.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.743+0000 7f4a8424e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:27.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.743+0000 7f4a8424e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:27.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.743+0000 7f4a8424e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:27.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.743+0000 7f4a8424e640 1 -- 172.21.15.170:0/2593924051 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a7c1264c0 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.746+0000 7f4a727fc640 1 -- 172.21.15.170:0/2593924051 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f4a7801a950 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.746+0000 7f4a817e9640 10 monclient: get_auth_request con 0x7f4a6006ddd0 auth_method 0 2024-03-29T11:18:27.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.746+0000 7f4a727fc640 1 -- 172.21.15.170:0/2593924051 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f4a78084830 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.746+0000 7f4a727fc640 10 client.4467.objecter ms_dispatch 0x7f4a7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:27.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.746+0000 7f4a727fc640 3 client.4467.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:27.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.746+0000 7f4a727fc640 3 client.4467.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:27.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.746+0000 7f4a727fc640 20 client.4467.objecter dump_active .. 0 homeless 2024-03-29T11:18:27.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.747+0000 7f4a727fc640 10 client.4467.objecter ms_handle_connect 0x7f4a6006ddd0 2024-03-29T11:18:27.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.753+0000 7f4a727fc640 1 -- 172.21.15.170:0/2593924051 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4a78050170 con 0x7f4a7c0ac5a0 2024-03-29T11:18:27.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.753+0000 7f4a727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:27.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:27.753+0000 7f4a727fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:28.079 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.103+0000 7f4a8424e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:28.080 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.103+0000 7f4a8424e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:28.080 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.103+0000 7f4a8424e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:28.080 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.103+0000 7f4a8424e640 1 -- 172.21.15.170:0/2593924051 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a7c075f10 con 0x7f4a7c0ac5a0 2024-03-29T11:18:28.080 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.104+0000 7f4a727fc640 1 -- 172.21.15.170:0/2593924051 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f4a780520a0 con 0x7f4a7c0ac5a0 2024-03-29T11:18:28.080 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.104+0000 7f4a727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:28.081 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.104+0000 7f4a727fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:28.081 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:18:28.081 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:28.084 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.107+0000 7f4a8424e640 10 librados: watch_flush enter 2024-03-29T11:18:28.085 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.108+0000 7f4a8424e640 10 librados: watch_flush exit 2024-03-29T11:18:28.085 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.108+0000 7f4a8424e640 20 client.4467.objecter shutdown clearing up homeless session... 2024-03-29T11:18:28.085 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.108+0000 7f4a8424e640 10 client.4467.objecter successfully canceled tick 2024-03-29T11:18:28.085 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.108+0000 7f4a8424e640 1 -- 172.21.15.170:0/2593924051 >> v1:172.21.15.170:6816/3298994444 conn(0x7f4a6006ddd0 legacy=0x7f4a60070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:28.085 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.108+0000 7f4a8424e640 10 monclient: shutdown 2024-03-29T11:18:28.085 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.108+0000 7f4a8424e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:28.085 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.108+0000 7f4a8424e640 1 -- 172.21.15.170:0/2593924051 >> v1:172.21.15.170:6789/0 conn(0x7f4a7c0ac5a0 legacy=0x7f4a7c144ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:28.085 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.108+0000 7f4a8424e640 1 -- 172.21.15.170:0/2593924051 shutdown_connections 2024-03-29T11:18:28.085 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.108+0000 7f4a8424e640 1 -- 172.21.15.170:0/2593924051 >> 172.21.15.170:0/2593924051 conn(0x7f4a7c06b660 msgr2=0x7f4a7c0b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:28.085 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.108+0000 7f4a8424e640 1 -- 172.21.15.170:0/2593924051 shutdown_connections 2024-03-29T11:18:28.085 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.108+0000 7f4a8424e640 1 -- 172.21.15.170:0/2593924051 wait complete. 2024-03-29T11:18:28.085 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:28.108+0000 7f4a8424e640 1 librados: shutdown 2024-03-29T11:18:28.100 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:34.907 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:35.069 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.092+0000 7f333964a640 10 monclient: get_monmap_and_config 2024-03-29T11:18:35.069 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.092+0000 7f333964a640 10 monclient: build_initial_monmap 2024-03-29T11:18:35.069 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.092+0000 7f333964a640 10 monclient: monmap: 2024-03-29T11:18:35.069 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:18:35.069 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:35.069 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:18:35.092732+0000 2024-03-29T11:18:35.069 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:18:35.092732+0000 2024-03-29T11:18:35.069 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:35.069 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:35.069 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:18:35.069 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:18:35.070 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:18:35.070 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:35.070 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.093+0000 7f333964a640 1 Processor -- start 2024-03-29T11:18:35.070 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.093+0000 7f333964a640 1 -- start start 2024-03-29T11:18:35.071 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.093+0000 7f333964a640 10 monclient: init 2024-03-29T11:18:35.071 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.094+0000 7f333964a640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:35.071 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.094+0000 7f333964a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:18:35.071 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.094+0000 7f333964a640 10 monclient(hunting): picked mon.noname-c con 0x7f33340b45a0 addr v1:172.21.15.184:6789/0 2024-03-29T11:18:35.071 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.094+0000 7f333964a640 10 monclient(hunting): picked mon.noname-a con 0x7f333413fb10 addr v1:172.21.15.170:6789/0 2024-03-29T11:18:35.071 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.094+0000 7f333964a640 10 monclient(hunting): picked mon.noname-b con 0x7f33340b3920 addr v1:172.21.15.170:6790/0 2024-03-29T11:18:35.071 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.094+0000 7f333964a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33340b1de0 con 0x7f333413fb10 2024-03-29T11:18:35.071 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.094+0000 7f333964a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33340b0670 con 0x7f33340b45a0 2024-03-29T11:18:35.071 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.094+0000 7f333964a640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33340af560 con 0x7f33340b3920 2024-03-29T11:18:35.071 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.094+0000 7f333964a640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:35.071 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.094+0000 7f333964a640 10 monclient(hunting): authenticate will time out at 1296.405491s 2024-03-29T11:18:35.071 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.094+0000 7f3332ffd640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f33340b45a0 0x7f333413f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:54260/0 (socket says 172.21.15.170:54260) 2024-03-29T11:18:35.071 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.094+0000 7f3332ffd640 1 -- 172.21.15.170:0/1052573268 learned_addr learned my addr 172.21.15.170:0/1052573268 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:18:35.072 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.095+0000 7f3331ffb640 1 -- 172.21.15.170:0/1052573268 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560425202 0 0) 0x7f33340af560 con 0x7f33340b3920 2024-03-29T11:18:35.072 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.095+0000 7f3331ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:35.072 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.095+0000 7f3331ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:35.072 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.095+0000 7f3331ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:35.072 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.095+0000 7f3331ffb640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:18:35.072 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.095+0000 7f3331ffb640 1 -- 172.21.15.170:0/1052573268 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f331c003520 con 0x7f33340b3920 2024-03-29T11:18:35.072 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.095+0000 7f3331ffb640 1 -- 172.21.15.170:0/1052573268 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2624519167 0 0) 0x7f33340b0670 con 0x7f33340b45a0 2024-03-29T11:18:35.072 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.095+0000 7f3331ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:35.072 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.095+0000 7f3331ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:35.072 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.095+0000 7f3331ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:35.072 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.095+0000 7f3331ffb640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:18:35.073 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.095+0000 7f3331ffb640 1 -- 172.21.15.170:0/1052573268 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33340af560 con 0x7f33340b45a0 2024-03-29T11:18:35.073 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 1 -- 172.21.15.170:0/1052573268 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588394298 0 0) 0x7f33340b1de0 con 0x7f333413fb10 2024-03-29T11:18:35.073 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:35.074 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:35.074 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:35.074 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:18:35.074 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 1 -- 172.21.15.170:0/1052573268 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33340b0670 con 0x7f333413fb10 2024-03-29T11:18:35.074 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 1 -- 172.21.15.170:0/1052573268 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 453249751 0 0) 0x7f33340af560 con 0x7f33340b45a0 2024-03-29T11:18:35.074 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 1 -- 172.21.15.170:0/1052573268 >> v1:172.21.15.170:6790/0 conn(0x7f33340b3920 legacy=0x7f33340b16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:35.074 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 1 -- 172.21.15.170:0/1052573268 >> v1:172.21.15.170:6789/0 conn(0x7f333413fb10 legacy=0x7f33340b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:35.074 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:35.074 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 1 monclient: found mon.noname-c 2024-03-29T11:18:35.074 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:35.074 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:18:35.074 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 1 -- 172.21.15.170:0/1052573268 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33340b08c0 con 0x7f33340b45a0 2024-03-29T11:18:35.074 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:35.075 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:35.075 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:35.075 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 1 -- 172.21.15.170:0/1052573268 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f33240015f0 con 0x7f33340b45a0 2024-03-29T11:18:35.075 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:35.075 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:35.075 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f3331ffb640 10 monclient: dump: 2024-03-29T11:18:35.075 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:35.075 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:35.075 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:35.075 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:35.075 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:35.075 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:35.075 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:35.075 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:35.075 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:35.076 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:35.076 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f333964a640 5 monclient: authenticate success, global_id 4462 2024-03-29T11:18:35.076 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.096+0000 7f333964a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:35.076 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.097+0000 7f3331ffb640 1 -- 172.21.15.170:0/1052573268 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3324001750 con 0x7f33340b45a0 2024-03-29T11:18:35.076 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.097+0000 7f3331ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:35.076 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.097+0000 7f333964a640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:35.076 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.097+0000 7f333964a640 4 set_mon_vals no callback set 2024-03-29T11:18:35.076 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.097+0000 7f3331ffb640 1 -- 172.21.15.170:0/1052573268 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3324001b60 con 0x7f33340b45a0 2024-03-29T11:18:35.076 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.097+0000 7f3331ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:35.076 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.097+0000 7f3331ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:35.076 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.097+0000 7f3331ffb640 10 monclient: dump: 2024-03-29T11:18:35.076 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:35.076 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:35.076 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:35.076 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:35.077 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:35.077 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:35.077 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:35.077 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:35.077 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:35.077 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:35.080 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.098+0000 7f333964a640 10 monclient: shutdown 2024-03-29T11:18:35.080 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.098+0000 7f333964a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:35.080 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.098+0000 7f333964a640 1 -- 172.21.15.170:0/1052573268 >> v1:172.21.15.184:6789/0 conn(0x7f33340b45a0 legacy=0x7f333413f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:35.080 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.098+0000 7f333964a640 1 -- 172.21.15.170:0/1052573268 shutdown_connections 2024-03-29T11:18:35.081 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.098+0000 7f333964a640 1 -- 172.21.15.170:0/1052573268 >> 172.21.15.170:0/1052573268 conn(0x7f3334066a00 msgr2=0x7f3334066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:35.081 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.098+0000 7f333964a640 1 -- 172.21.15.170:0/1052573268 shutdown_connections 2024-03-29T11:18:35.081 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.098+0000 7f333964a640 1 -- 172.21.15.170:0/1052573268 wait complete. 2024-03-29T11:18:35.081 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.099+0000 7f333964a640 10 monclient: build_initial_monmap 2024-03-29T11:18:35.081 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.099+0000 7f333964a640 10 monclient: monmap: 2024-03-29T11:18:35.081 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:18:35.081 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:35.081 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:18:35.081 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:18:35.081 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:35.081 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:35.081 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:18:35.081 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:18:35.081 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:18:35.082 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:35.082 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.099+0000 7f333964a640 1 librados: starting msgr at 2024-03-29T11:18:35.082 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.099+0000 7f333964a640 1 librados: starting objecter 2024-03-29T11:18:35.082 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.100+0000 7f333964a640 1 Processor -- start 2024-03-29T11:18:35.082 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.100+0000 7f333964a640 1 -- start start 2024-03-29T11:18:35.082 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.100+0000 7f333964a640 1 librados: setting wanted keys 2024-03-29T11:18:35.082 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.100+0000 7f333964a640 1 librados: calling monclient init 2024-03-29T11:18:35.082 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.100+0000 7f333964a640 10 monclient: init 2024-03-29T11:18:35.082 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.101+0000 7f333964a640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:35.082 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.101+0000 7f333964a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:35.082 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.101+0000 7f333964a640 10 monclient(hunting): picked mon.noname-b con 0x7f33340b3920 addr v1:172.21.15.184:6789/0 2024-03-29T11:18:35.082 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.101+0000 7f333964a640 10 monclient(hunting): picked mon.noname-c con 0x7f33340b45a0 addr v1:172.21.15.170:6790/0 2024-03-29T11:18:35.082 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.101+0000 7f333964a640 10 monclient(hunting): picked mon.noname-a con 0x7f333413fb10 addr v1:172.21.15.170:6789/0 2024-03-29T11:18:35.082 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.101+0000 7f3332ffd640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f33340b3920 0x7f3334070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:54476/0 (socket says 172.21.15.170:54476) 2024-03-29T11:18:35.083 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.101+0000 7f3332ffd640 1 -- 172.21.15.170:0/2769295275 learned_addr learned my addr 172.21.15.170:0/2769295275 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:18:35.083 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.101+0000 7f333964a640 1 -- 172.21.15.170:0/2769295275 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3334070110 con 0x7f333413fb10 2024-03-29T11:18:35.083 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.101+0000 7f331b7fe640 10 client.?.objecter ms_handle_connect 0x7f33340b3920 2024-03-29T11:18:35.083 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.101+0000 7f331b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:35.083 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.101+0000 7f333964a640 1 -- 172.21.15.170:0/2769295275 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3334071500 con 0x7f33340b3920 2024-03-29T11:18:35.083 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.101+0000 7f333964a640 1 -- 172.21.15.170:0/2769295275 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3334071690 con 0x7f33340b45a0 2024-03-29T11:18:35.083 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.101+0000 7f333964a640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:35.083 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.101+0000 7f333964a640 10 monclient(hunting): authenticate will time out at 1296.412643s 2024-03-29T11:18:35.083 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.101+0000 7f331b7fe640 10 client.?.objecter ms_handle_connect 0x7f333413fb10 2024-03-29T11:18:35.083 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.101+0000 7f331b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:35.083 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 1 -- 172.21.15.170:0/2769295275 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875966249 0 0) 0x7f3334070110 con 0x7f333413fb10 2024-03-29T11:18:35.083 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:35.083 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:35.083 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:35.084 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:18:35.084 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 1 -- 172.21.15.170:0/2769295275 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3320004eb0 con 0x7f333413fb10 2024-03-29T11:18:35.084 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 1 -- 172.21.15.170:0/2769295275 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137225104 0 0) 0x7f3334071500 con 0x7f33340b3920 2024-03-29T11:18:35.084 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:35.084 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:35.084 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:35.084 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:18:35.084 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 1 -- 172.21.15.170:0/2769295275 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3334070110 con 0x7f33340b3920 2024-03-29T11:18:35.084 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 10 client.?.objecter ms_handle_connect 0x7f33340b45a0 2024-03-29T11:18:35.084 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:35.084 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 1 -- 172.21.15.170:0/2769295275 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1711101259 0 0) 0x7f3334070110 con 0x7f33340b3920 2024-03-29T11:18:35.084 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 1 -- 172.21.15.170:0/2769295275 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3334071500 con 0x7f33340b3920 2024-03-29T11:18:35.084 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 1 -- 172.21.15.170:0/2769295275 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3324001aa0 con 0x7f33340b3920 2024-03-29T11:18:35.084 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:35.085 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:35.085 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 10 monclient(hunting): dump: 2024-03-29T11:18:35.085 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:35.085 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:35.085 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:35.085 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:35.085 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:35.085 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:35.085 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:35.085 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:35.085 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:35.085 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:35.085 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:35.085 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 1 -- 172.21.15.170:0/2769295275 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2544133481 0 0) 0x7f3334071500 con 0x7f33340b3920 2024-03-29T11:18:35.085 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 1 -- 172.21.15.170:0/2769295275 >> v1:172.21.15.170:6790/0 conn(0x7f33340b45a0 legacy=0x7f33341d2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:35.086 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 1 -- 172.21.15.170:0/2769295275 >> v1:172.21.15.170:6789/0 conn(0x7f333413fb10 legacy=0x7f33341dd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:35.086 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:18:35.086 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 1 monclient: found mon.b 2024-03-29T11:18:35.086 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:35.086 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:18:35.086 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.102+0000 7f331b7fe640 1 -- 172.21.15.170:0/2769295275 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33341d3470 con 0x7f33340b3920 2024-03-29T11:18:35.086 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.103+0000 7f331b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:18:35.086 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.103+0000 7f331b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:18:35.086 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.103+0000 7f331b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:35.086 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.103+0000 7f333964a640 5 monclient: authenticate success, global_id 4465 2024-03-29T11:18:35.086 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.103+0000 7f333964a640 10 monclient: _renew_subs 2024-03-29T11:18:35.086 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.103+0000 7f333964a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:18:35.086 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.103+0000 7f333964a640 1 -- 172.21.15.170:0/2769295275 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33340b8680 con 0x7f33340b3920 2024-03-29T11:18:35.086 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.103+0000 7f333964a640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:35.087 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.103+0000 7f333964a640 10 monclient: _renew_subs 2024-03-29T11:18:35.087 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.103+0000 7f333964a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:18:35.087 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.103+0000 7f333964a640 1 -- 172.21.15.170:0/2769295275 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3334072030 con 0x7f33340b3920 2024-03-29T11:18:35.087 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.103+0000 7f333964a640 1 librados: init done 2024-03-29T11:18:35.087 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.103+0000 7f331b7fe640 1 -- 172.21.15.170:0/2769295275 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3324004a90 con 0x7f33340b3920 2024-03-29T11:18:35.087 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.103+0000 7f331b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:35.087 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.103+0000 7f331b7fe640 1 -- 172.21.15.170:0/2769295275 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3324004da0 con 0x7f33340b3920 2024-03-29T11:18:35.087 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.103+0000 7f331b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:35.087 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.103+0000 7f331b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:35.087 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.103+0000 7f331b7fe640 10 monclient: dump: 2024-03-29T11:18:35.087 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:35.087 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:35.087 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:35.087 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:35.087 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:35.088 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:35.088 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:35.088 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:35.088 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:35.088 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:35.088 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.103+0000 7f33317fa640 4 set_mon_vals no callback set 2024-03-29T11:18:35.089 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.104+0000 7f333964a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:35.089 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.104+0000 7f333964a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:35.089 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.104+0000 7f333964a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:18:35.089 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.104+0000 7f333964a640 1 -- 172.21.15.170:0/2769295275 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3314005080 con 0x7f33340b3920 2024-03-29T11:18:35.089 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.106+0000 7f331b7fe640 1 -- 172.21.15.170:0/2769295275 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f332401ad70 con 0x7f33340b3920 2024-03-29T11:18:35.089 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.107+0000 7f331b7fe640 1 -- 172.21.15.170:0/2769295275 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f3324084ae0 con 0x7f33340b3920 2024-03-29T11:18:35.089 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.107+0000 7f331b7fe640 10 client.4465.objecter ms_dispatch 0x7f3334000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:35.090 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.107+0000 7f331b7fe640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:35.090 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.107+0000 7f331b7fe640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:35.090 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.107+0000 7f331b7fe640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-29T11:18:35.090 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.107+0000 7f33327fc640 10 monclient: get_auth_request con 0x7f332006dd60 auth_method 0 2024-03-29T11:18:35.090 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.108+0000 7f331b7fe640 10 client.4465.objecter ms_handle_connect 0x7f332006dd60 2024-03-29T11:18:35.090 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.112+0000 7f331b7fe640 1 -- 172.21.15.170:0/2769295275 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33240503c0 con 0x7f33340b3920 2024-03-29T11:18:35.090 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.113+0000 7f331b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:35.090 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.113+0000 7f331b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:35.526 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.550+0000 7f333964a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:35.526 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.550+0000 7f333964a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:35.526 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.550+0000 7f333964a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:18:35.526 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.550+0000 7f333964a640 1 -- 172.21.15.170:0/2769295275 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3314005330 con 0x7f33340b3920 2024-03-29T11:18:35.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.551+0000 7f331b7fe640 1 -- 172.21.15.170:0/2769295275 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f33240522f0 con 0x7f33340b3920 2024-03-29T11:18:35.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.551+0000 7f331b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:35.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.551+0000 7f331b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:35.528 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:18:35.528 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:35.532 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.555+0000 7f333964a640 10 librados: watch_flush enter 2024-03-29T11:18:35.532 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.555+0000 7f333964a640 10 librados: watch_flush exit 2024-03-29T11:18:35.532 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.555+0000 7f333964a640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-29T11:18:35.532 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.555+0000 7f333964a640 10 client.4465.objecter successfully canceled tick 2024-03-29T11:18:35.533 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.555+0000 7f333964a640 1 -- 172.21.15.170:0/2769295275 >> v1:172.21.15.170:6816/3298994444 conn(0x7f332006dd60 legacy=0x7f3320070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:35.533 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.555+0000 7f333964a640 10 monclient: shutdown 2024-03-29T11:18:35.533 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.555+0000 7f333964a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:35.533 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.555+0000 7f333964a640 1 -- 172.21.15.170:0/2769295275 >> v1:172.21.15.184:6789/0 conn(0x7f33340b3920 legacy=0x7f3334070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:35.533 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.556+0000 7f333964a640 1 -- 172.21.15.170:0/2769295275 shutdown_connections 2024-03-29T11:18:35.533 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.556+0000 7f333964a640 1 -- 172.21.15.170:0/2769295275 >> 172.21.15.170:0/2769295275 conn(0x7f3334066a00 msgr2=0x7f33341c8580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:35.533 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.556+0000 7f333964a640 1 -- 172.21.15.170:0/2769295275 shutdown_connections 2024-03-29T11:18:35.533 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.556+0000 7f333964a640 1 -- 172.21.15.170:0/2769295275 wait complete. 2024-03-29T11:18:35.533 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:35.556+0000 7f333964a640 1 librados: shutdown 2024-03-29T11:18:35.549 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:42.356 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:42.491 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.512+0000 7fea9e1cf640 10 monclient: get_monmap_and_config 2024-03-29T11:18:42.491 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.512+0000 7fea9e1cf640 10 monclient: build_initial_monmap 2024-03-29T11:18:42.491 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.513+0000 7fea9e1cf640 10 monclient: monmap: 2024-03-29T11:18:42.491 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:18:42.491 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:42.491 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:18:42.514458+0000 2024-03-29T11:18:42.491 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:18:42.514458+0000 2024-03-29T11:18:42.491 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:42.492 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:42.492 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:18:42.492 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:18:42.492 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:18:42.492 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:42.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.514+0000 7fea9e1cf640 1 Processor -- start 2024-03-29T11:18:42.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.514+0000 7fea9e1cf640 1 -- start start 2024-03-29T11:18:42.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.514+0000 7fea9e1cf640 10 monclient: init 2024-03-29T11:18:42.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.514+0000 7fea9e1cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:42.502 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.515+0000 7fea9e1cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:18:42.503 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.515+0000 7fea9e1cf640 10 monclient(hunting): picked mon.noname-c con 0x7fea980b0500 addr v1:172.21.15.184:6789/0 2024-03-29T11:18:42.503 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.515+0000 7fea9e1cf640 10 monclient(hunting): picked mon.noname-a con 0x7fea980b08e0 addr v1:172.21.15.170:6789/0 2024-03-29T11:18:42.503 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.515+0000 7fea9e1cf640 10 monclient(hunting): picked mon.noname-b con 0x7fea980aeca0 addr v1:172.21.15.170:6790/0 2024-03-29T11:18:42.503 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.515+0000 7fea9e1cf640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea98140620 con 0x7fea980b08e0 2024-03-29T11:18:42.503 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.515+0000 7fea9e1cf640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea9813daf0 con 0x7fea980b0500 2024-03-29T11:18:42.503 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.515+0000 7fea9e1cf640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea980ae180 con 0x7fea980aeca0 2024-03-29T11:18:42.503 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.515+0000 7fea9e1cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:42.503 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.515+0000 7fea9e1cf640 10 monclient(hunting): authenticate will time out at 1303.827119s 2024-03-29T11:18:42.503 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.515+0000 7fea977fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fea980b0500 0x7fea9813ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:33322/0 (socket says 172.21.15.170:33322) 2024-03-29T11:18:42.504 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.515+0000 7fea96ffd640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fea980b08e0 0x7fea9813d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:41842/0 (socket says 172.21.15.170:41842) 2024-03-29T11:18:42.504 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.515+0000 7fea96ffd640 1 -- 172.21.15.170:0/1946860302 learned_addr learned my addr 172.21.15.170:0/1946860302 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:18:42.504 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.515+0000 7fea967fc640 1 -- 172.21.15.170:0/1946860302 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968665740 0 0) 0x7fea98140620 con 0x7fea980b08e0 2024-03-29T11:18:42.504 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.515+0000 7fea967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:42.504 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.515+0000 7fea967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:42.504 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.515+0000 7fea967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:42.504 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.515+0000 7fea967fc640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:18:42.504 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.516+0000 7fea967fc640 1 -- 172.21.15.170:0/1946860302 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea80003520 con 0x7fea980b08e0 2024-03-29T11:18:42.504 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.516+0000 7fea967fc640 1 -- 172.21.15.170:0/1946860302 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2272046741 0 0) 0x7fea9813daf0 con 0x7fea980b0500 2024-03-29T11:18:42.504 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.516+0000 7fea967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:42.504 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.516+0000 7fea967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:42.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.516+0000 7fea967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:42.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.516+0000 7fea967fc640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:18:42.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.516+0000 7fea967fc640 1 -- 172.21.15.170:0/1946860302 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea98140620 con 0x7fea980b0500 2024-03-29T11:18:42.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.516+0000 7fea967fc640 1 -- 172.21.15.170:0/1946860302 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 127622925 0 0) 0x7fea80003520 con 0x7fea980b08e0 2024-03-29T11:18:42.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.516+0000 7fea967fc640 1 -- 172.21.15.170:0/1946860302 >> v1:172.21.15.170:6790/0 conn(0x7fea980aeca0 legacy=0x7fea9814b7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:42.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea967fc640 1 -- 172.21.15.170:0/1946860302 >> v1:172.21.15.184:6789/0 conn(0x7fea980b0500 legacy=0x7fea9813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:42.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea967fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:42.505 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea967fc640 1 monclient: found mon.noname-a 2024-03-29T11:18:42.506 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:42.506 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea967fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:18:42.506 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea967fc640 1 -- 172.21.15.170:0/1946860302 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea980accf0 con 0x7fea980b08e0 2024-03-29T11:18:42.506 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea967fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:42.506 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea967fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:42.506 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:42.506 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea967fc640 1 -- 172.21.15.170:0/1946860302 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fea8c001690 con 0x7fea980b08e0 2024-03-29T11:18:42.506 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:42.506 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:42.507 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea967fc640 10 monclient: dump: 2024-03-29T11:18:42.507 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:42.507 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:42.507 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:42.507 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:42.507 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:42.507 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:42.507 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:42.507 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:42.507 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:42.508 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:42.508 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea9e1cf640 5 monclient: authenticate success, global_id 4476 2024-03-29T11:18:42.508 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea9e1cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:42.508 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea967fc640 1 -- 172.21.15.170:0/1946860302 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea8c0017f0 con 0x7fea980b08e0 2024-03-29T11:18:42.508 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:42.508 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea967fc640 1 -- 172.21.15.170:0/1946860302 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fea8c001c00 con 0x7fea980b08e0 2024-03-29T11:18:42.508 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:42.508 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:42.508 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea967fc640 10 monclient: dump: 2024-03-29T11:18:42.508 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:42.508 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:42.508 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:42.509 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:42.509 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:42.509 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:42.509 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:42.509 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:42.509 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:42.509 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:42.509 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea9e1cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:42.509 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.517+0000 7fea9e1cf640 4 set_mon_vals no callback set 2024-03-29T11:18:42.509 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.519+0000 7fea9e1cf640 10 monclient: shutdown 2024-03-29T11:18:42.509 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.519+0000 7fea9e1cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:42.509 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.519+0000 7fea9e1cf640 1 -- 172.21.15.170:0/1946860302 >> v1:172.21.15.170:6789/0 conn(0x7fea980b08e0 legacy=0x7fea9813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:42.509 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.519+0000 7fea9e1cf640 1 -- 172.21.15.170:0/1946860302 shutdown_connections 2024-03-29T11:18:42.509 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.519+0000 7fea9e1cf640 1 -- 172.21.15.170:0/1946860302 >> 172.21.15.170:0/1946860302 conn(0x7fea9806b660 msgr2=0x7fea9806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:42.510 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.519+0000 7fea9e1cf640 1 -- 172.21.15.170:0/1946860302 shutdown_connections 2024-03-29T11:18:42.510 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.519+0000 7fea9e1cf640 1 -- 172.21.15.170:0/1946860302 wait complete. 2024-03-29T11:18:42.510 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.519+0000 7fea9e1cf640 10 monclient: build_initial_monmap 2024-03-29T11:18:42.510 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.519+0000 7fea9e1cf640 10 monclient: monmap: 2024-03-29T11:18:42.510 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:18:42.510 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:42.510 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:18:42.510 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:18:42.510 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:42.510 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:42.510 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:18:42.510 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:18:42.510 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:18:42.511 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:42.511 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.520+0000 7fea9e1cf640 1 librados: starting msgr at 2024-03-29T11:18:42.511 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.520+0000 7fea9e1cf640 1 librados: starting objecter 2024-03-29T11:18:42.511 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.520+0000 7fea9e1cf640 1 Processor -- start 2024-03-29T11:18:42.511 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.520+0000 7fea9e1cf640 1 -- start start 2024-03-29T11:18:42.511 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.520+0000 7fea9e1cf640 1 librados: setting wanted keys 2024-03-29T11:18:42.511 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.520+0000 7fea9e1cf640 1 librados: calling monclient init 2024-03-29T11:18:42.511 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.520+0000 7fea9e1cf640 10 monclient: init 2024-03-29T11:18:42.511 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.521+0000 7fea9e1cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:42.511 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.521+0000 7fea9e1cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:18:42.511 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.521+0000 7fea9e1cf640 10 monclient(hunting): picked mon.noname-b con 0x7fea980aeca0 addr v1:172.21.15.184:6789/0 2024-03-29T11:18:42.512 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.521+0000 7fea9e1cf640 10 monclient(hunting): picked mon.noname-a con 0x7fea980b0500 addr v1:172.21.15.170:6789/0 2024-03-29T11:18:42.512 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.521+0000 7fea9e1cf640 10 monclient(hunting): picked mon.noname-c con 0x7fea981c8c60 addr v1:172.21.15.170:6790/0 2024-03-29T11:18:42.512 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.521+0000 7fea9e1cf640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea981db870 con 0x7fea980b0500 2024-03-29T11:18:42.512 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.521+0000 7fea9e1cf640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea981cb050 con 0x7fea980aeca0 2024-03-29T11:18:42.512 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.521+0000 7fea9e1cf640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea981c6480 con 0x7fea981c8c60 2024-03-29T11:18:42.512 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.521+0000 7fea9e1cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:42.512 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.521+0000 7fea9e1cf640 10 monclient(hunting): authenticate will time out at 1303.833834s 2024-03-29T11:18:42.512 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.522+0000 7fea96ffd640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fea980b0500 0x7fea981dbd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:43740/0 (socket says 172.21.15.170:43740) 2024-03-29T11:18:42.512 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.522+0000 7fea96ffd640 1 -- 172.21.15.170:0/2588526271 learned_addr learned my addr 172.21.15.170:0/2588526271 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:18:42.512 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.522+0000 7fea7ffff640 10 client.?.objecter ms_handle_connect 0x7fea980b0500 2024-03-29T11:18:42.512 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.522+0000 7fea7ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:42.512 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.522+0000 7fea7ffff640 1 -- 172.21.15.170:0/2588526271 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974852335 0 0) 0x7fea981db870 con 0x7fea980b0500 2024-03-29T11:18:42.512 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.522+0000 7fea7ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:42.513 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.522+0000 7fea7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:42.513 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.522+0000 7fea7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:42.513 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.522+0000 7fea7ffff640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:18:42.513 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.522+0000 7fea7ffff640 1 -- 172.21.15.170:0/2588526271 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea84004eb0 con 0x7fea980b0500 2024-03-29T11:18:42.513 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.523+0000 7fea7ffff640 1 -- 172.21.15.170:0/2588526271 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1215242477 0 0) 0x7fea84004eb0 con 0x7fea980b0500 2024-03-29T11:18:42.513 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.523+0000 7fea7ffff640 1 -- 172.21.15.170:0/2588526271 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea981db870 con 0x7fea980b0500 2024-03-29T11:18:42.513 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.523+0000 7fea7ffff640 1 -- 172.21.15.170:0/2588526271 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fea8c001c40 con 0x7fea980b0500 2024-03-29T11:18:42.513 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.523+0000 7fea7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:42.513 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.523+0000 7fea7ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:42.513 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.523+0000 7fea7ffff640 10 monclient(hunting): dump: 2024-03-29T11:18:42.513 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:42.513 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:42.513 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:42.514 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:42.514 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:42.514 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:42.514 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:42.514 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:42.514 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:42.514 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:42.514 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.523+0000 7fea7ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:42.514 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.523+0000 7fea7ffff640 1 -- 172.21.15.170:0/2588526271 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 55474154 0 0) 0x7fea981db870 con 0x7fea980b0500 2024-03-29T11:18:42.514 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.523+0000 7fea7ffff640 1 -- 172.21.15.170:0/2588526271 >> v1:172.21.15.170:6790/0 conn(0x7fea981c8c60 legacy=0x7fea981e6320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:18:42.519 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea7ffff640 1 -- 172.21.15.170:0/2588526271 >> v1:172.21.15.184:6789/0 conn(0x7fea980aeca0 legacy=0x7fea981c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:42.519 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea7ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:42.519 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea7ffff640 1 monclient: found mon.a 2024-03-29T11:18:42.519 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:42.519 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea7ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:42.519 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea7ffff640 1 -- 172.21.15.170:0/2588526271 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea981dc410 con 0x7fea980b0500 2024-03-29T11:18:42.519 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea7ffff640 10 monclient: _finish_auth 0 2024-03-29T11:18:42.519 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea7ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:42.519 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:42.519 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea9e1cf640 5 monclient: authenticate success, global_id 4479 2024-03-29T11:18:42.520 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea9e1cf640 10 monclient: _renew_subs 2024-03-29T11:18:42.520 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea9e1cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:42.520 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea9e1cf640 1 -- 172.21.15.170:0/2588526271 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea981dc6c0 con 0x7fea980b0500 2024-03-29T11:18:42.520 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea9e1cf640 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:42.520 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea9e1cf640 10 monclient: _renew_subs 2024-03-29T11:18:42.520 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea9e1cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:42.520 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea9e1cf640 1 -- 172.21.15.170:0/2588526271 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea981e7c20 con 0x7fea980b0500 2024-03-29T11:18:42.520 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea9e1cf640 1 librados: init done 2024-03-29T11:18:42.520 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea7ffff640 1 -- 172.21.15.170:0/2588526271 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea8c004a90 con 0x7fea980b0500 2024-03-29T11:18:42.520 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea7ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:42.520 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea7ffff640 1 -- 172.21.15.170:0/2588526271 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fea8c004ee0 con 0x7fea980b0500 2024-03-29T11:18:42.520 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:42.520 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea7ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:42.521 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea7ffff640 10 monclient: dump: 2024-03-29T11:18:42.521 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:42.521 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:42.521 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:42.521 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:42.521 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:42.521 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:42.521 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:42.521 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:42.521 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:42.521 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:42.521 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.525+0000 7fea95ffb640 4 set_mon_vals no callback set 2024-03-29T11:18:42.521 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.527+0000 7fea7ffff640 1 -- 172.21.15.170:0/2588526271 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fea8c005040 con 0x7fea980b0500 2024-03-29T11:18:42.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.528+0000 7fea977fe640 10 monclient: get_auth_request con 0x7fea8406dc90 auth_method 0 2024-03-29T11:18:42.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.528+0000 7fea7ffff640 1 -- 172.21.15.170:0/2588526271 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fea8c084bf0 con 0x7fea980b0500 2024-03-29T11:18:42.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.528+0000 7fea7ffff640 10 client.4479.objecter ms_dispatch 0x7fea98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:42.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.528+0000 7fea7ffff640 3 client.4479.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:42.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.528+0000 7fea7ffff640 3 client.4479.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:42.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.528+0000 7fea7ffff640 20 client.4479.objecter dump_active .. 0 homeless 2024-03-29T11:18:42.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.528+0000 7fea7ffff640 10 client.4479.objecter ms_handle_connect 0x7fea8406dc90 2024-03-29T11:18:42.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.529+0000 7fea9e1cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:42.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.529+0000 7fea9e1cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:42.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.529+0000 7fea9e1cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:42.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.529+0000 7fea9e1cf640 1 -- 172.21.15.170:0/2588526271 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea78005080 con 0x7fea980b0500 2024-03-29T11:18:42.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.535+0000 7fea7ffff640 1 -- 172.21.15.170:0/2588526271 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea8c0504d0 con 0x7fea980b0500 2024-03-29T11:18:42.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.536+0000 7fea7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:42.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.536+0000 7fea7ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:42.822 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.845+0000 7fea9e1cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:42.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.845+0000 7fea9e1cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:42.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.845+0000 7fea9e1cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:18:42.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.845+0000 7fea9e1cf640 1 -- 172.21.15.170:0/2588526271 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea78005330 con 0x7fea980b0500 2024-03-29T11:18:42.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.847+0000 7fea7ffff640 1 -- 172.21.15.170:0/2588526271 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fea8c052400 con 0x7fea980b0500 2024-03-29T11:18:42.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.847+0000 7fea7ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:42.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.847+0000 7fea7ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:42.825 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:18:42.825 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:42.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.850+0000 7fea9e1cf640 10 librados: watch_flush enter 2024-03-29T11:18:42.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.850+0000 7fea9e1cf640 10 librados: watch_flush exit 2024-03-29T11:18:42.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.850+0000 7fea9e1cf640 20 client.4479.objecter shutdown clearing up homeless session... 2024-03-29T11:18:42.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.850+0000 7fea9e1cf640 10 client.4479.objecter successfully canceled tick 2024-03-29T11:18:42.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.850+0000 7fea9e1cf640 1 -- 172.21.15.170:0/2588526271 >> v1:172.21.15.170:6816/3298994444 conn(0x7fea8406dc90 legacy=0x7fea84070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:42.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.850+0000 7fea9e1cf640 10 monclient: shutdown 2024-03-29T11:18:42.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.850+0000 7fea9e1cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:42.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.850+0000 7fea9e1cf640 1 -- 172.21.15.170:0/2588526271 >> v1:172.21.15.170:6789/0 conn(0x7fea980b0500 legacy=0x7fea981dbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:42.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.850+0000 7fea9e1cf640 1 -- 172.21.15.170:0/2588526271 shutdown_connections 2024-03-29T11:18:42.829 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.850+0000 7fea9e1cf640 1 -- 172.21.15.170:0/2588526271 >> 172.21.15.170:0/2588526271 conn(0x7fea9806b660 msgr2=0x7fea980a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:42.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.853+0000 7fea9e1cf640 1 -- 172.21.15.170:0/2588526271 shutdown_connections 2024-03-29T11:18:42.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.853+0000 7fea9e1cf640 1 -- 172.21.15.170:0/2588526271 wait complete. 2024-03-29T11:18:42.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:42.854+0000 7fea9e1cf640 1 librados: shutdown 2024-03-29T11:18:42.850 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:49.657 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:49.789 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.811+0000 7fa5fc9b3640 10 monclient: get_monmap_and_config 2024-03-29T11:18:49.790 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.811+0000 7fa5fc9b3640 10 monclient: build_initial_monmap 2024-03-29T11:18:49.790 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.811+0000 7fa5fc9b3640 10 monclient: monmap: 2024-03-29T11:18:49.790 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:18:49.790 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:49.790 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:18:49.813216+0000 2024-03-29T11:18:49.790 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:18:49.813216+0000 2024-03-29T11:18:49.790 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:49.790 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:49.790 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:18:49.790 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:18:49.790 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:18:49.790 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:49.791 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.813+0000 7fa5fc9b3640 1 Processor -- start 2024-03-29T11:18:49.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.813+0000 7fa5fc9b3640 1 -- start start 2024-03-29T11:18:49.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.813+0000 7fa5fc9b3640 10 monclient: init 2024-03-29T11:18:49.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.814+0000 7fa5fc9b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:49.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.814+0000 7fa5fc9b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:49.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.814+0000 7fa5fc9b3640 10 monclient(hunting): picked mon.noname-a con 0x7fa5f0006800 addr v1:172.21.15.170:6789/0 2024-03-29T11:18:49.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.814+0000 7fa5fc9b3640 10 monclient(hunting): picked mon.noname-b con 0x7fa5f0006be0 addr v1:172.21.15.170:6790/0 2024-03-29T11:18:49.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.814+0000 7fa5fc9b3640 10 monclient(hunting): picked mon.noname-c con 0x7fa5f0031b60 addr v1:172.21.15.184:6789/0 2024-03-29T11:18:49.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.814+0000 7fa5fc9b3640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f0029370 con 0x7fa5f0006800 2024-03-29T11:18:49.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.814+0000 7fa5fc9b3640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f00295e0 con 0x7fa5f0031b60 2024-03-29T11:18:49.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.814+0000 7fa5f6ffd640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fa5f0006be0 0x7fa5f00c4ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:40068/0 (socket says 172.21.15.170:40068) 2024-03-29T11:18:49.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.814+0000 7fa5f6ffd640 1 -- 172.21.15.170:0/2410757611 learned_addr learned my addr 172.21.15.170:0/2410757611 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:18:49.792 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.814+0000 7fa5fc9b3640 1 -- 172.21.15.170:0/2410757611 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f0031360 con 0x7fa5f0006be0 2024-03-29T11:18:49.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.815+0000 7fa5fc9b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:49.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.815+0000 7fa5fc9b3640 10 monclient(hunting): authenticate will time out at 1311.127089s 2024-03-29T11:18:49.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.815+0000 7fa5f67fc640 1 -- 172.21.15.170:0/2410757611 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157814742 0 0) 0x7fa5f0031360 con 0x7fa5f0006be0 2024-03-29T11:18:49.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.815+0000 7fa5f67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:49.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.815+0000 7fa5f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:49.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.815+0000 7fa5f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:49.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.815+0000 7fa5f67fc640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:18:49.793 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.815+0000 7fa5f67fc640 1 -- 172.21.15.170:0/2410757611 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5e8003520 con 0x7fa5f0006be0 2024-03-29T11:18:49.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.815+0000 7fa5f67fc640 1 -- 172.21.15.170:0/2410757611 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1948647675 0 0) 0x7fa5e8003520 con 0x7fa5f0006be0 2024-03-29T11:18:49.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.815+0000 7fa5f67fc640 1 -- 172.21.15.170:0/2410757611 >> v1:172.21.15.184:6789/0 conn(0x7fa5f0031b60 legacy=0x7fa5f00d30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:49.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.815+0000 7fa5f67fc640 1 -- 172.21.15.170:0/2410757611 >> v1:172.21.15.170:6789/0 conn(0x7fa5f0006800 legacy=0x7fa5f00cad20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:49.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.817+0000 7fa5f67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:49.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.817+0000 7fa5f67fc640 1 monclient: found mon.noname-b 2024-03-29T11:18:49.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.817+0000 7fa5f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:49.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.817+0000 7fa5f67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:18:49.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.817+0000 7fa5f67fc640 1 -- 172.21.15.170:0/2410757611 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f0031640 con 0x7fa5f0006be0 2024-03-29T11:18:49.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.817+0000 7fa5f67fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:49.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.817+0000 7fa5f67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:49.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.817+0000 7fa5f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:49.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.817+0000 7fa5f67fc640 1 -- 172.21.15.170:0/2410757611 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa5e4001670 con 0x7fa5f0006be0 2024-03-29T11:18:49.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.817+0000 7fa5f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:49.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.817+0000 7fa5f67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:49.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.817+0000 7fa5f67fc640 10 monclient: dump: 2024-03-29T11:18:49.804 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:49.804 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:49.804 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:49.804 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:49.804 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:49.804 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:49.804 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:49.804 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:49.804 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:49.805 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:49.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.818+0000 7fa5fc9b3640 5 monclient: authenticate success, global_id 4511 2024-03-29T11:18:49.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.818+0000 7fa5fc9b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:49.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.818+0000 7fa5f67fc640 1 -- 172.21.15.170:0/2410757611 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5e40018d0 con 0x7fa5f0006be0 2024-03-29T11:18:49.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.818+0000 7fa5f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:49.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.818+0000 7fa5fc9b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:49.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.818+0000 7fa5fc9b3640 4 set_mon_vals no callback set 2024-03-29T11:18:49.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.818+0000 7fa5f67fc640 1 -- 172.21.15.170:0/2410757611 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa5e4001ca0 con 0x7fa5f0006be0 2024-03-29T11:18:49.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.819+0000 7fa5f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:49.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.819+0000 7fa5f67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:49.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.819+0000 7fa5f67fc640 10 monclient: dump: 2024-03-29T11:18:49.806 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:49.806 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:49.806 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:49.806 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:49.806 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:49.806 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:49.806 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:49.806 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:49.806 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:49.806 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:49.806 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.819+0000 7fa5fc9b3640 10 monclient: shutdown 2024-03-29T11:18:49.806 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.819+0000 7fa5fc9b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:49.806 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.819+0000 7fa5fc9b3640 1 -- 172.21.15.170:0/2410757611 >> v1:172.21.15.170:6790/0 conn(0x7fa5f0006be0 legacy=0x7fa5f00c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:49.806 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.819+0000 7fa5fc9b3640 1 -- 172.21.15.170:0/2410757611 shutdown_connections 2024-03-29T11:18:49.807 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.819+0000 7fa5fc9b3640 1 -- 172.21.15.170:0/2410757611 >> 172.21.15.170:0/2410757611 conn(0x7fa5f00058f0 msgr2=0x7fa5f0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:49.807 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.819+0000 7fa5fc9b3640 1 -- 172.21.15.170:0/2410757611 shutdown_connections 2024-03-29T11:18:49.807 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.819+0000 7fa5fc9b3640 1 -- 172.21.15.170:0/2410757611 wait complete. 2024-03-29T11:18:49.807 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.820+0000 7fa5fc9b3640 10 monclient: build_initial_monmap 2024-03-29T11:18:49.807 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.820+0000 7fa5fc9b3640 10 monclient: monmap: 2024-03-29T11:18:49.807 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:18:49.807 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:49.807 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:18:49.807 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:18:49.808 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:49.808 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:49.808 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:18:49.808 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:18:49.808 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:18:49.808 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:49.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.821+0000 7fa5fc9b3640 1 librados: starting msgr at 2024-03-29T11:18:49.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.821+0000 7fa5fc9b3640 1 librados: starting objecter 2024-03-29T11:18:49.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.821+0000 7fa5fc9b3640 1 Processor -- start 2024-03-29T11:18:49.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.822+0000 7fa5fc9b3640 1 -- start start 2024-03-29T11:18:49.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.822+0000 7fa5fc9b3640 1 librados: setting wanted keys 2024-03-29T11:18:49.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.822+0000 7fa5fc9b3640 1 librados: calling monclient init 2024-03-29T11:18:49.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.822+0000 7fa5fc9b3640 10 monclient: init 2024-03-29T11:18:49.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5fc9b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:49.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5fc9b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:49.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5fc9b3640 10 monclient(hunting): picked mon.noname-a con 0x7fa5f0006800 addr v1:172.21.15.170:6789/0 2024-03-29T11:18:49.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5fc9b3640 10 monclient(hunting): picked mon.noname-c con 0x7fa5f0031b60 addr v1:172.21.15.170:6790/0 2024-03-29T11:18:49.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5f6ffd640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fa5f0031b60 0x7fa5f00bcf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:42366/0 (socket says 172.21.15.170:42366) 2024-03-29T11:18:49.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5f6ffd640 1 -- 172.21.15.170:0/2371995177 learned_addr learned my addr 172.21.15.170:0/2371995177 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:18:49.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5fc9b3640 10 monclient(hunting): picked mon.noname-b con 0x7fa5f00bd620 addr v1:172.21.15.184:6789/0 2024-03-29T11:18:49.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5e3fff640 10 client.?.objecter ms_handle_connect 0x7fa5f0031b60 2024-03-29T11:18:49.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:49.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5fc9b3640 1 -- 172.21.15.170:0/2371995177 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f00c6870 con 0x7fa5f0006800 2024-03-29T11:18:49.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5fc9b3640 1 -- 172.21.15.170:0/2371995177 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f00c55a0 con 0x7fa5f00bd620 2024-03-29T11:18:49.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5fc9b3640 1 -- 172.21.15.170:0/2371995177 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f015c980 con 0x7fa5f0031b60 2024-03-29T11:18:49.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5fc9b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:49.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5fc9b3640 10 monclient(hunting): authenticate will time out at 1311.135630s 2024-03-29T11:18:49.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5e3fff640 1 -- 172.21.15.170:0/2371995177 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548674688 0 0) 0x7fa5f015c980 con 0x7fa5f0031b60 2024-03-29T11:18:49.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:49.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:49.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:49.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5e3fff640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:18:49.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.823+0000 7fa5e3fff640 1 -- 172.21.15.170:0/2371995177 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5dc004eb0 con 0x7fa5f0031b60 2024-03-29T11:18:49.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.824+0000 7fa5e3fff640 1 -- 172.21.15.170:0/2371995177 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 255505668 0 0) 0x7fa5dc004eb0 con 0x7fa5f0031b60 2024-03-29T11:18:49.810 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.824+0000 7fa5e3fff640 1 -- 172.21.15.170:0/2371995177 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5f015c980 con 0x7fa5f0031b60 2024-03-29T11:18:49.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.824+0000 7fa5e3fff640 1 -- 172.21.15.170:0/2371995177 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa5e4001be0 con 0x7fa5f0031b60 2024-03-29T11:18:49.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.824+0000 7fa5e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:49.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.824+0000 7fa5e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:49.811 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.824+0000 7fa5e3fff640 10 monclient(hunting): dump: 2024-03-29T11:18:49.811 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:49.811 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:49.811 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:49.811 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:49.811 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:49.811 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:49.811 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:49.812 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:49.812 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:49.812 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:49.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.824+0000 7fa5e3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:49.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.824+0000 7fa5e3fff640 1 -- 172.21.15.170:0/2371995177 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2075793660 0 0) 0x7fa5f015c980 con 0x7fa5f0031b60 2024-03-29T11:18:49.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.824+0000 7fa5e3fff640 1 -- 172.21.15.170:0/2371995177 >> v1:172.21.15.184:6789/0 conn(0x7fa5f00bd620 legacy=0x7fa5f015cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:49.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.824+0000 7fa5e3fff640 1 -- 172.21.15.170:0/2371995177 >> v1:172.21.15.170:6789/0 conn(0x7fa5f0006800 legacy=0x7fa5f00c6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:18:49.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.825+0000 7fa5e3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:49.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.825+0000 7fa5e3fff640 1 monclient: found mon.c 2024-03-29T11:18:49.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.825+0000 7fa5e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:49.812 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.825+0000 7fa5e3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:18:49.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.825+0000 7fa5e3fff640 1 -- 172.21.15.170:0/2371995177 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f00c21b0 con 0x7fa5f0031b60 2024-03-29T11:18:49.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.825+0000 7fa5e3fff640 10 monclient: _finish_auth 0 2024-03-29T11:18:49.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.825+0000 7fa5e3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:49.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.825+0000 7fa5e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:49.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.825+0000 7fa5fc9b3640 5 monclient: authenticate success, global_id 4514 2024-03-29T11:18:49.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.825+0000 7fa5fc9b3640 10 monclient: _renew_subs 2024-03-29T11:18:49.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.825+0000 7fa5fc9b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:18:49.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.825+0000 7fa5fc9b3640 1 -- 172.21.15.170:0/2371995177 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f015d5b0 con 0x7fa5f0031b60 2024-03-29T11:18:49.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.825+0000 7fa5fc9b3640 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:49.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.825+0000 7fa5fc9b3640 10 monclient: _renew_subs 2024-03-29T11:18:49.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.825+0000 7fa5fc9b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:18:49.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.825+0000 7fa5fc9b3640 1 -- 172.21.15.170:0/2371995177 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f00c6ed0 con 0x7fa5f0031b60 2024-03-29T11:18:49.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.825+0000 7fa5fc9b3640 1 librados: init done 2024-03-29T11:18:49.816 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.827+0000 7fa5e3fff640 1 -- 172.21.15.170:0/2371995177 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5e40011d0 con 0x7fa5f0031b60 2024-03-29T11:18:49.816 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.827+0000 7fa5e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:49.816 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.827+0000 7fa5e3fff640 1 -- 172.21.15.170:0/2371995177 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa5e4004d90 con 0x7fa5f0031b60 2024-03-29T11:18:49.816 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.827+0000 7fa5e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:49.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.827+0000 7fa5e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:49.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.827+0000 7fa5e3fff640 10 monclient: dump: 2024-03-29T11:18:49.817 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:49.817 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:49.817 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:49.817 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:49.817 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:49.817 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:49.817 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:49.817 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:49.817 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:49.817 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:49.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.827+0000 7fa5fc9b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:49.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.827+0000 7fa5fc9b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:49.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.827+0000 7fa5fc9b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:18:49.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.827+0000 7fa5fc9b3640 1 -- 172.21.15.170:0/2371995177 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f0004cd0 con 0x7fa5f0031b60 2024-03-29T11:18:49.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.828+0000 7fa5f5ffb640 4 set_mon_vals no callback set 2024-03-29T11:18:49.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.828+0000 7fa5e3fff640 1 -- 172.21.15.170:0/2371995177 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fa5e401ada0 con 0x7fa5f0031b60 2024-03-29T11:18:49.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.828+0000 7fa5e3fff640 1 -- 172.21.15.170:0/2371995177 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fa5e4084d20 con 0x7fa5f0031b60 2024-03-29T11:18:49.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.828+0000 7fa5e3fff640 10 client.4514.objecter ms_dispatch 0x7fa5f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:49.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.829+0000 7fa5e3fff640 3 client.4514.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:49.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.829+0000 7fa5e3fff640 3 client.4514.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:49.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.829+0000 7fa5e3fff640 20 client.4514.objecter dump_active .. 0 homeless 2024-03-29T11:18:49.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.829+0000 7fa5f77fe640 10 monclient: get_auth_request con 0x7fa5dc06dbb0 auth_method 0 2024-03-29T11:18:49.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.829+0000 7fa5e3fff640 10 client.4514.objecter ms_handle_connect 0x7fa5dc06dbb0 2024-03-29T11:18:49.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.836+0000 7fa5e3fff640 1 -- 172.21.15.170:0/2371995177 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5e4050430 con 0x7fa5f0031b60 2024-03-29T11:18:49.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.836+0000 7fa5e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:49.819 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:49.836+0000 7fa5e3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:50.127 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.149+0000 7fa5fc9b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:50.127 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.149+0000 7fa5fc9b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:50.127 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.149+0000 7fa5fc9b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:18:50.127 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.149+0000 7fa5fc9b3640 1 -- 172.21.15.170:0/2371995177 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5f00086f0 con 0x7fa5f0031b60 2024-03-29T11:18:50.128 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.150+0000 7fa5e3fff640 1 -- 172.21.15.170:0/2371995177 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fa5e4052530 con 0x7fa5f0031b60 2024-03-29T11:18:50.128 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.150+0000 7fa5e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:50.128 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.150+0000 7fa5e3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:50.128 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:18:50.128 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:50.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.154+0000 7fa5fc9b3640 10 librados: watch_flush enter 2024-03-29T11:18:50.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.154+0000 7fa5fc9b3640 10 librados: watch_flush exit 2024-03-29T11:18:50.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.154+0000 7fa5fc9b3640 20 client.4514.objecter shutdown clearing up homeless session... 2024-03-29T11:18:50.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.154+0000 7fa5fc9b3640 10 client.4514.objecter successfully canceled tick 2024-03-29T11:18:50.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.154+0000 7fa5fc9b3640 1 -- 172.21.15.170:0/2371995177 >> v1:172.21.15.170:6816/3298994444 conn(0x7fa5dc06dbb0 legacy=0x7fa5dc070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:50.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.154+0000 7fa5fc9b3640 10 monclient: shutdown 2024-03-29T11:18:50.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.154+0000 7fa5fc9b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:50.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.154+0000 7fa5fc9b3640 1 -- 172.21.15.170:0/2371995177 >> v1:172.21.15.170:6790/0 conn(0x7fa5f0031b60 legacy=0x7fa5f00bcf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:50.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.154+0000 7fa5fc9b3640 1 -- 172.21.15.170:0/2371995177 shutdown_connections 2024-03-29T11:18:50.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.154+0000 7fa5fc9b3640 1 -- 172.21.15.170:0/2371995177 >> 172.21.15.170:0/2371995177 conn(0x7fa5f00058f0 msgr2=0x7fa5f002a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:50.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.155+0000 7fa5fc9b3640 1 -- 172.21.15.170:0/2371995177 shutdown_connections 2024-03-29T11:18:50.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.155+0000 7fa5fc9b3640 1 -- 172.21.15.170:0/2371995177 wait complete. 2024-03-29T11:18:50.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:50.155+0000 7fa5fc9b3640 1 librados: shutdown 2024-03-29T11:18:50.147 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:56.954 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:57.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.127+0000 7f3b9f5e9640 10 monclient: get_monmap_and_config 2024-03-29T11:18:57.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.127+0000 7f3b9f5e9640 10 monclient: build_initial_monmap 2024-03-29T11:18:57.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.127+0000 7f3b9f5e9640 10 monclient: monmap: 2024-03-29T11:18:57.106 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:18:57.106 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:57.106 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:18:57.129140+0000 2024-03-29T11:18:57.106 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:18:57.129140+0000 2024-03-29T11:18:57.106 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:57.106 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:57.107 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:18:57.107 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:18:57.107 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:18:57.107 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:57.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.130+0000 7f3b9f5e9640 1 Processor -- start 2024-03-29T11:18:57.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.130+0000 7f3b9f5e9640 1 -- start start 2024-03-29T11:18:57.108 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.130+0000 7f3b9f5e9640 10 monclient: init 2024-03-29T11:18:57.108 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.131+0000 7f3b9f5e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:57.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.131+0000 7f3b9f5e9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:57.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.131+0000 7f3b9f5e9640 10 monclient(hunting): picked mon.noname-a con 0x7f3b980a9ac0 addr v1:172.21.15.170:6789/0 2024-03-29T11:18:57.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.131+0000 7f3b9f5e9640 10 monclient(hunting): picked mon.noname-b con 0x7f3b9813ee90 addr v1:172.21.15.170:6790/0 2024-03-29T11:18:57.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.131+0000 7f3b9f5e9640 10 monclient(hunting): picked mon.noname-c con 0x7f3b980af900 addr v1:172.21.15.184:6789/0 2024-03-29T11:18:57.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.131+0000 7f3b9f5e9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b980afd10 con 0x7f3b980a9ac0 2024-03-29T11:18:57.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.131+0000 7f3b9f5e9640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b980af6f0 con 0x7f3b980af900 2024-03-29T11:18:57.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.131+0000 7f3b9f5e9640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b980aac80 con 0x7f3b9813ee90 2024-03-29T11:18:57.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.131+0000 7f3b9f5e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:57.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.131+0000 7f3b9f5e9640 10 monclient(hunting): authenticate will time out at 1318.443513s 2024-03-29T11:18:57.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.131+0000 7f3b9cb84640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f3b9813ee90 0x7f3b9813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:49216/0 (socket says 172.21.15.170:49216) 2024-03-29T11:18:57.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.131+0000 7f3b9cb84640 1 -- 172.21.15.170:0/3814236985 learned_addr learned my addr 172.21.15.170:0/3814236985 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:18:57.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 1 -- 172.21.15.170:0/3814236985 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541414696 0 0) 0x7f3b980aac80 con 0x7f3b9813ee90 2024-03-29T11:18:57.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:18:57.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 1 -- 172.21.15.170:0/3814236985 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b84003520 con 0x7f3b9813ee90 2024-03-29T11:18:57.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 1 -- 172.21.15.170:0/3814236985 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1441094248 0 0) 0x7f3b84003520 con 0x7f3b9813ee90 2024-03-29T11:18:57.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 1 -- 172.21.15.170:0/3814236985 >> v1:172.21.15.184:6789/0 conn(0x7f3b980af900 legacy=0x7f3b98149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 1 -- 172.21.15.170:0/3814236985 >> v1:172.21.15.170:6789/0 conn(0x7f3b980a9ac0 legacy=0x7f3b980adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:57.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:57.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 1 monclient: found mon.noname-b 2024-03-29T11:18:57.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:57.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:18:57.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 1 -- 172.21.15.170:0/3814236985 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b980aaf60 con 0x7f3b9813ee90 2024-03-29T11:18:57.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 10 monclient: _finish_auth 0 2024-03-29T11:18:57.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:57.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:57.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b9f5e9640 5 monclient: authenticate success, global_id 4517 2024-03-29T11:18:57.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b9f5e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:57.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 1 -- 172.21.15.170:0/3814236985 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3b88001690 con 0x7f3b9813ee90 2024-03-29T11:18:57.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:57.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b8ffff640 10 monclient: dump: 2024-03-29T11:18:57.120 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:57.120 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:57.120 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:57.120 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:57.120 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.120 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:57.120 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:57.120 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:57.120 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:57.121 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:57.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.135+0000 7f3b9f5e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:57.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.136+0000 7f3b8ffff640 1 -- 172.21.15.170:0/3814236985 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b88001860 con 0x7f3b9813ee90 2024-03-29T11:18:57.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.136+0000 7f3b8ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:57.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.136+0000 7f3b9f5e9640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:57.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.136+0000 7f3b9f5e9640 4 set_mon_vals no callback set 2024-03-29T11:18:57.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.136+0000 7f3b8ffff640 1 -- 172.21.15.170:0/3814236985 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3b88001cb0 con 0x7f3b9813ee90 2024-03-29T11:18:57.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.136+0000 7f3b8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.136+0000 7f3b8ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:57.121 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.136+0000 7f3b8ffff640 10 monclient: dump: 2024-03-29T11:18:57.122 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:57.122 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:57.122 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:57.122 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:57.122 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.122 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:57.122 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:57.122 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:57.122 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:57.123 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:57.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.137+0000 7f3b9f5e9640 10 monclient: shutdown 2024-03-29T11:18:57.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.137+0000 7f3b9f5e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:57.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.137+0000 7f3b9f5e9640 1 -- 172.21.15.170:0/3814236985 >> v1:172.21.15.170:6790/0 conn(0x7f3b9813ee90 legacy=0x7f3b9813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.137+0000 7f3b9f5e9640 1 -- 172.21.15.170:0/3814236985 shutdown_connections 2024-03-29T11:18:57.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.137+0000 7f3b9f5e9640 1 -- 172.21.15.170:0/3814236985 >> 172.21.15.170:0/3814236985 conn(0x7f3b980b35e0 msgr2=0x7f3b980b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:57.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.137+0000 7f3b9f5e9640 1 -- 172.21.15.170:0/3814236985 shutdown_connections 2024-03-29T11:18:57.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.137+0000 7f3b9f5e9640 1 -- 172.21.15.170:0/3814236985 wait complete. 2024-03-29T11:18:57.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.138+0000 7f3b9f5e9640 10 monclient: build_initial_monmap 2024-03-29T11:18:57.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.138+0000 7f3b9f5e9640 10 monclient: monmap: 2024-03-29T11:18:57.124 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:18:57.124 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:57.124 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:18:57.124 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:18:57.124 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:57.124 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:57.124 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:18:57.124 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:18:57.125 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:18:57.125 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:57.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.139+0000 7f3b9f5e9640 1 librados: starting msgr at 2024-03-29T11:18:57.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.139+0000 7f3b9f5e9640 1 librados: starting objecter 2024-03-29T11:18:57.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.139+0000 7f3b9f5e9640 1 Processor -- start 2024-03-29T11:18:57.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.139+0000 7f3b9f5e9640 1 -- start start 2024-03-29T11:18:57.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.139+0000 7f3b9f5e9640 1 librados: setting wanted keys 2024-03-29T11:18:57.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.139+0000 7f3b9f5e9640 1 librados: calling monclient init 2024-03-29T11:18:57.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.139+0000 7f3b9f5e9640 10 monclient: init 2024-03-29T11:18:57.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.140+0000 7f3b9f5e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:57.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.140+0000 7f3b9f5e9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:18:57.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.140+0000 7f3b9f5e9640 10 monclient(hunting): picked mon.noname-c con 0x7f3b980a9ac0 addr v1:172.21.15.170:6790/0 2024-03-29T11:18:57.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.140+0000 7f3b9f5e9640 10 monclient(hunting): picked mon.noname-a con 0x7f3b980af900 addr v1:172.21.15.170:6789/0 2024-03-29T11:18:57.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.140+0000 7f3b9f5e9640 10 monclient(hunting): picked mon.noname-b con 0x7f3b9813ee90 addr v1:172.21.15.184:6789/0 2024-03-29T11:18:57.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.140+0000 7f3b9f5e9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b980704e0 con 0x7f3b980af900 2024-03-29T11:18:57.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.140+0000 7f3b9f5e9640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9806f490 con 0x7f3b9813ee90 2024-03-29T11:18:57.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.140+0000 7f3b9f5e9640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b98070710 con 0x7f3b980a9ac0 2024-03-29T11:18:57.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.140+0000 7f3b9f5e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:57.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.140+0000 7f3b9f5e9640 10 monclient(hunting): authenticate will time out at 1318.452150s 2024-03-29T11:18:57.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.142+0000 7f3b9d385640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f3b980a9ac0 0x7f3b981398f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:50236/0 (socket says 172.21.15.170:50236) 2024-03-29T11:18:57.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.142+0000 7f3b9d385640 1 -- 172.21.15.170:0/1481548684 learned_addr learned my addr 172.21.15.170:0/1481548684 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:18:57.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.142+0000 7f3b8dffb640 10 client.?.objecter ms_handle_connect 0x7f3b980a9ac0 2024-03-29T11:18:57.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.142+0000 7f3b8dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:57.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.142+0000 7f3b8dffb640 1 -- 172.21.15.170:0/1481548684 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753826966 0 0) 0x7f3b98070710 con 0x7f3b980a9ac0 2024-03-29T11:18:57.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.142+0000 7f3b8dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.142+0000 7f3b8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.142+0000 7f3b8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.142+0000 7f3b8dffb640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:18:57.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.142+0000 7f3b8dffb640 1 -- 172.21.15.170:0/1481548684 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b7c004eb0 con 0x7f3b980a9ac0 2024-03-29T11:18:57.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.143+0000 7f3b8dffb640 1 -- 172.21.15.170:0/1481548684 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2676058255 0 0) 0x7f3b7c004eb0 con 0x7f3b980a9ac0 2024-03-29T11:18:57.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.143+0000 7f3b8dffb640 1 -- 172.21.15.170:0/1481548684 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b98070710 con 0x7f3b980a9ac0 2024-03-29T11:18:57.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.143+0000 7f3b8dffb640 1 -- 172.21.15.170:0/1481548684 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3b90004400 con 0x7f3b980a9ac0 2024-03-29T11:18:57.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.143+0000 7f3b8dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.143+0000 7f3b8dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:57.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.143+0000 7f3b8dffb640 10 monclient(hunting): dump: 2024-03-29T11:18:57.135 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:57.135 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:57.135 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:57.135 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:57.135 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.135 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:57.135 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:57.136 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:57.136 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:57.136 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:57.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.143+0000 7f3b8dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:57.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.146+0000 7f3b8dffb640 10 client.?.objecter ms_handle_connect 0x7f3b980af900 2024-03-29T11:18:57.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.146+0000 7f3b8dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:57.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.146+0000 7f3b8dffb640 1 -- 172.21.15.170:0/1481548684 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4247811777 0 0) 0x7f3b98070710 con 0x7f3b980a9ac0 2024-03-29T11:18:57.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.146+0000 7f3b8dffb640 1 -- 172.21.15.170:0/1481548684 >> v1:172.21.15.184:6789/0 conn(0x7f3b9813ee90 legacy=0x7f3b981d6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.146+0000 7f3b8dffb640 1 -- 172.21.15.170:0/1481548684 >> v1:172.21.15.170:6789/0 conn(0x7f3b980af900 legacy=0x7f3b981d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.146+0000 7f3b8dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:57.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.146+0000 7f3b8dffb640 1 monclient: found mon.c 2024-03-29T11:18:57.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.146+0000 7f3b8dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:57.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.146+0000 7f3b8dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:18:57.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.146+0000 7f3b8dffb640 1 -- 172.21.15.170:0/1481548684 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9806ef70 con 0x7f3b980a9ac0 2024-03-29T11:18:57.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.146+0000 7f3b8dffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:57.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.146+0000 7f3b8dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:57.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.146+0000 7f3b8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:57.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.146+0000 7f3b9f5e9640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:18:57.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.146+0000 7f3b9f5e9640 10 monclient: _renew_subs 2024-03-29T11:18:57.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.146+0000 7f3b9f5e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:18:57.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.146+0000 7f3b9f5e9640 1 -- 172.21.15.170:0/1481548684 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b9806f640 con 0x7f3b980a9ac0 2024-03-29T11:18:57.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.147+0000 7f3b9f5e9640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:57.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.147+0000 7f3b9f5e9640 10 monclient: _renew_subs 2024-03-29T11:18:57.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.147+0000 7f3b9f5e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:18:57.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.147+0000 7f3b9f5e9640 1 -- 172.21.15.170:0/1481548684 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b9813a1e0 con 0x7f3b980a9ac0 2024-03-29T11:18:57.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.147+0000 7f3b9f5e9640 1 librados: init done 2024-03-29T11:18:57.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.147+0000 7f3b8dffb640 1 -- 172.21.15.170:0/1481548684 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b90004660 con 0x7f3b980a9ac0 2024-03-29T11:18:57.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.147+0000 7f3b8dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:57.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.147+0000 7f3b8f7fe640 4 set_mon_vals no callback set 2024-03-29T11:18:57.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.149+0000 7f3b8dffb640 1 -- 172.21.15.170:0/1481548684 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3b900016a0 con 0x7f3b980a9ac0 2024-03-29T11:18:57.138 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.149+0000 7f3b8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.138 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.149+0000 7f3b8dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:57.138 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.149+0000 7f3b8dffb640 10 monclient: dump: 2024-03-29T11:18:57.138 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:18:57.138 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:18:57.138 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:57.138 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:18:57.138 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.138 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:18:57.138 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:18:57.138 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:18:57.139 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:18:57.139 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:18:57.139 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.149+0000 7f3b8dffb640 1 -- 172.21.15.170:0/1481548684 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f3b900018c0 con 0x7f3b980a9ac0 2024-03-29T11:18:57.139 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.150+0000 7f3b8dffb640 1 -- 172.21.15.170:0/1481548684 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f3b90084f60 con 0x7f3b980a9ac0 2024-03-29T11:18:57.139 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.150+0000 7f3b8dffb640 10 client.4520.objecter ms_dispatch 0x7f3b98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:57.139 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.150+0000 7f3b8dffb640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:57.139 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.150+0000 7f3b8dffb640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:57.139 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.150+0000 7f3b8dffb640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-29T11:18:57.139 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.150+0000 7f3b9cb84640 10 monclient: get_auth_request con 0x7f3b7c06ddd0 auth_method 0 2024-03-29T11:18:57.139 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.151+0000 7f3b8dffb640 10 client.4520.objecter ms_handle_connect 0x7f3b7c06ddd0 2024-03-29T11:18:57.139 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.151+0000 7f3b9f5e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:57.139 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.151+0000 7f3b9f5e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:57.139 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.152+0000 7f3b9f5e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:18:57.139 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.152+0000 7f3b9f5e9640 1 -- 172.21.15.170:0/1481548684 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b78005080 con 0x7f3b980a9ac0 2024-03-29T11:18:57.140 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.162+0000 7f3b8dffb640 1 -- 172.21.15.170:0/1481548684 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3b900508c0 con 0x7f3b980a9ac0 2024-03-29T11:18:57.140 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.162+0000 7f3b8dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:57.140 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.162+0000 7f3b8dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:57.467 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.490+0000 7f3b9f5e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:57.467 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.490+0000 7f3b9f5e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:57.467 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.490+0000 7f3b9f5e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:18:57.467 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.490+0000 7f3b9f5e9640 1 -- 172.21.15.170:0/1481548684 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b78005330 con 0x7f3b980a9ac0 2024-03-29T11:18:57.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.492+0000 7f3b8dffb640 1 -- 172.21.15.170:0/1481548684 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f3b900527f0 con 0x7f3b980a9ac0 2024-03-29T11:18:57.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.492+0000 7f3b8dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:57.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.492+0000 7f3b8dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:57.471 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:18:57.471 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:57.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.497+0000 7f3b9f5e9640 10 librados: watch_flush enter 2024-03-29T11:18:57.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.497+0000 7f3b9f5e9640 10 librados: watch_flush exit 2024-03-29T11:18:57.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.497+0000 7f3b9f5e9640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-29T11:18:57.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.497+0000 7f3b9f5e9640 10 client.4520.objecter successfully canceled tick 2024-03-29T11:18:57.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.497+0000 7f3b9f5e9640 1 -- 172.21.15.170:0/1481548684 >> v1:172.21.15.170:6816/3298994444 conn(0x7f3b7c06ddd0 legacy=0x7f3b7c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.497+0000 7f3b9f5e9640 10 monclient: shutdown 2024-03-29T11:18:57.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.497+0000 7f3b9f5e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:57.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.497+0000 7f3b9f5e9640 1 -- 172.21.15.170:0/1481548684 >> v1:172.21.15.170:6790/0 conn(0x7f3b980a9ac0 legacy=0x7f3b981398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.501+0000 7f3b9f5e9640 1 -- 172.21.15.170:0/1481548684 shutdown_connections 2024-03-29T11:18:57.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.501+0000 7f3b9f5e9640 1 -- 172.21.15.170:0/1481548684 >> 172.21.15.170:0/1481548684 conn(0x7f3b980b35e0 msgr2=0x7f3b9813b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:57.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.501+0000 7f3b9f5e9640 1 -- 172.21.15.170:0/1481548684 shutdown_connections 2024-03-29T11:18:57.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.501+0000 7f3b9f5e9640 1 -- 172.21.15.170:0/1481548684 wait complete. 2024-03-29T11:18:57.479 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:18:57.501+0000 7f3b9f5e9640 1 librados: shutdown 2024-03-29T11:18:57.495 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:04.302 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:04.460 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.481+0000 7f5b32e8b640 10 monclient: get_monmap_and_config 2024-03-29T11:19:04.462 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.481+0000 7f5b32e8b640 10 monclient: build_initial_monmap 2024-03-29T11:19:04.462 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.481+0000 7f5b32e8b640 10 monclient: monmap: 2024-03-29T11:19:04.462 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:19:04.462 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:04.462 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:19:04.481543+0000 2024-03-29T11:19:04.463 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:19:04.481543+0000 2024-03-29T11:19:04.463 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:04.463 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:04.463 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:19:04.463 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:19:04.463 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:19:04.463 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:04.463 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.482+0000 7f5b32e8b640 1 Processor -- start 2024-03-29T11:19:04.463 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.482+0000 7f5b32e8b640 1 -- start start 2024-03-29T11:19:04.463 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.482+0000 7f5b32e8b640 10 monclient: init 2024-03-29T11:19:04.463 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.483+0000 7f5b32e8b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:04.463 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.483+0000 7f5b32e8b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:04.463 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.483+0000 7f5b32e8b640 10 monclient(hunting): picked mon.noname-a con 0x7f5b24029f00 addr v1:172.21.15.170:6789/0 2024-03-29T11:19:04.463 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.483+0000 7f5b32e8b640 10 monclient(hunting): picked mon.noname-b con 0x7f5b2402b090 addr v1:172.21.15.170:6790/0 2024-03-29T11:19:04.464 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.483+0000 7f5b32e8b640 10 monclient(hunting): picked mon.noname-c con 0x7f5b240e2c40 addr v1:172.21.15.184:6789/0 2024-03-29T11:19:04.464 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.483+0000 7f5b32e8b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2402c0f0 con 0x7f5b24029f00 2024-03-29T11:19:04.464 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.483+0000 7f5b32e8b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2402a430 con 0x7f5b240e2c40 2024-03-29T11:19:04.464 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.483+0000 7f5b32e8b640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2402a5c0 con 0x7f5b2402b090 2024-03-29T11:19:04.464 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.483+0000 7f5b32e8b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:04.464 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.483+0000 7f5b32e8b640 10 monclient(hunting): authenticate will time out at 1325.794428s 2024-03-29T11:19:04.464 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.486+0000 7f5b31e89640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f5b24029f00 0x7f5b240e8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:52731/0 (socket says 172.21.15.170:52731) 2024-03-29T11:19:04.464 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.486+0000 7f5b31e89640 1 -- 172.21.15.170:0/1936351175 learned_addr learned my addr 172.21.15.170:0/1936351175 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:19:04.468 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 1 -- 172.21.15.170:0/1936351175 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381398406 0 0) 0x7f5b2402a5c0 con 0x7f5b2402b090 2024-03-29T11:19:04.468 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.468 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.468 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.468 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:19:04.468 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 1 -- 172.21.15.170:0/1936351175 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b18003520 con 0x7f5b2402b090 2024-03-29T11:19:04.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 1 -- 172.21.15.170:0/1936351175 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585641263 0 0) 0x7f5b2402c0f0 con 0x7f5b24029f00 2024-03-29T11:19:04.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:19:04.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 1 -- 172.21.15.170:0/1936351175 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b2402a5c0 con 0x7f5b24029f00 2024-03-29T11:19:04.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 1 -- 172.21.15.170:0/1936351175 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1330728352 0 0) 0x7f5b2402a5c0 con 0x7f5b24029f00 2024-03-29T11:19:04.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 1 -- 172.21.15.170:0/1936351175 >> v1:172.21.15.170:6790/0 conn(0x7f5b2402b090 legacy=0x7f5b240e2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 1 -- 172.21.15.170:0/1936351175 >> v1:172.21.15.184:6789/0 conn(0x7f5b240e2c40 legacy=0x7f5b240f0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:04.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 10 monclient: _finish_hunting 0 2024-03-29T11:19:04.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 1 monclient: found mon.noname-a 2024-03-29T11:19:04.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:04.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:19:04.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 1 -- 172.21.15.170:0/1936351175 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b24006940 con 0x7f5b24029f00 2024-03-29T11:19:04.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 10 monclient: _finish_auth 0 2024-03-29T11:19:04.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 10 monclient: _check_auth_tickets 2024-03-29T11:19:04.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:04.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 1 -- 172.21.15.170:0/1936351175 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5b2c04f730 con 0x7f5b24029f00 2024-03-29T11:19:04.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:04.470 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 10 monclient: dump: 2024-03-29T11:19:04.470 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:04.470 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:04.470 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:04.470 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:04.470 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.470 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:04.470 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:04.470 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:04.471 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:04.471 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:04.471 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 1 -- 172.21.15.170:0/1936351175 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b2c0008d0 con 0x7f5b24029f00 2024-03-29T11:19:04.471 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:04.471 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 1 -- 172.21.15.170:0/1936351175 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5b2c0758b0 con 0x7f5b24029f00 2024-03-29T11:19:04.471 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.471 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:04.471 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b30e87640 10 monclient: dump: 2024-03-29T11:19:04.471 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:04.471 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:04.471 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:04.471 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:04.471 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.471 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:04.472 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:04.472 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:04.472 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:04.472 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:04.472 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.490+0000 7f5b32e8b640 5 monclient: authenticate success, global_id 4485 2024-03-29T11:19:04.472 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.491+0000 7f5b32e8b640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:04.472 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.491+0000 7f5b32e8b640 4 set_mon_vals no callback set 2024-03-29T11:19:04.472 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.491+0000 7f5b32e8b640 10 monclient: shutdown 2024-03-29T11:19:04.473 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.491+0000 7f5b32e8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:04.473 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.491+0000 7f5b32e8b640 1 -- 172.21.15.170:0/1936351175 >> v1:172.21.15.170:6789/0 conn(0x7f5b24029f00 legacy=0x7f5b240e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.473 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.493+0000 7f5b32e8b640 1 -- 172.21.15.170:0/1936351175 shutdown_connections 2024-03-29T11:19:04.473 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.493+0000 7f5b32e8b640 1 -- 172.21.15.170:0/1936351175 >> 172.21.15.170:0/1936351175 conn(0x7f5b240058f0 msgr2=0x7f5b24005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:04.473 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.495+0000 7f5b32e8b640 1 -- 172.21.15.170:0/1936351175 shutdown_connections 2024-03-29T11:19:04.473 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.497+0000 7f5b32e8b640 1 -- 172.21.15.170:0/1936351175 wait complete. 2024-03-29T11:19:04.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.497+0000 7f5b32e8b640 10 monclient: build_initial_monmap 2024-03-29T11:19:04.475 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.497+0000 7f5b32e8b640 10 monclient: monmap: 2024-03-29T11:19:04.475 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:19:04.475 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:04.475 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:19:04.475 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:19:04.475 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:04.475 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:04.476 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:19:04.477 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:19:04.477 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:19:04.477 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:04.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.498+0000 7f5b32e8b640 1 librados: starting msgr at 2024-03-29T11:19:04.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.498+0000 7f5b32e8b640 1 librados: starting objecter 2024-03-29T11:19:04.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.498+0000 7f5b32e8b640 1 Processor -- start 2024-03-29T11:19:04.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.498+0000 7f5b32e8b640 1 -- start start 2024-03-29T11:19:04.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.498+0000 7f5b32e8b640 1 librados: setting wanted keys 2024-03-29T11:19:04.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.498+0000 7f5b32e8b640 1 librados: calling monclient init 2024-03-29T11:19:04.477 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.498+0000 7f5b32e8b640 10 monclient: init 2024-03-29T11:19:04.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.499+0000 7f5b32e8b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:04.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.499+0000 7f5b32e8b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:04.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.499+0000 7f5b32e8b640 10 monclient(hunting): picked mon.noname-c con 0x7f5b24029f00 addr v1:172.21.15.170:6790/0 2024-03-29T11:19:04.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.499+0000 7f5b32e8b640 10 monclient(hunting): picked mon.noname-b con 0x7f5b2402b090 addr v1:172.21.15.184:6789/0 2024-03-29T11:19:04.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.499+0000 7f5b32e8b640 10 monclient(hunting): picked mon.noname-a con 0x7f5b240e2c40 addr v1:172.21.15.170:6789/0 2024-03-29T11:19:04.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.499+0000 7f5b32e8b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b24006e90 con 0x7f5b240e2c40 2024-03-29T11:19:04.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.499+0000 7f5b32e8b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b24037d90 con 0x7f5b2402b090 2024-03-29T11:19:04.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.499+0000 7f5b32e8b640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240ddcc0 con 0x7f5b24029f00 2024-03-29T11:19:04.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.499+0000 7f5b32e8b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:04.478 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.499+0000 7f5b32e8b640 10 monclient(hunting): authenticate will time out at 1325.810376s 2024-03-29T11:19:04.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.505+0000 7f5b31e89640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f5b24029f00 0x7f5b24036b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:56555/0 (socket says 172.21.15.170:56555) 2024-03-29T11:19:04.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.505+0000 7f5b31e89640 1 -- 172.21.15.170:0/3663213054 learned_addr learned my addr 172.21.15.170:0/3663213054 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:19:04.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 client.?.objecter ms_handle_connect 0x7f5b24029f00 2024-03-29T11:19:04.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:04.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 client.?.objecter ms_handle_connect 0x7f5b240e2c40 2024-03-29T11:19:04.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:04.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 1 -- 172.21.15.170:0/3663213054 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239415281 0 0) 0x7f5b240ddcc0 con 0x7f5b24029f00 2024-03-29T11:19:04.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:19:04.483 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 1 -- 172.21.15.170:0/3663213054 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b10004eb0 con 0x7f5b24029f00 2024-03-29T11:19:04.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 1 -- 172.21.15.170:0/3663213054 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3732829497 0 0) 0x7f5b10004eb0 con 0x7f5b24029f00 2024-03-29T11:19:04.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 1 -- 172.21.15.170:0/3663213054 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b10005300 con 0x7f5b24029f00 2024-03-29T11:19:04.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 1 -- 172.21.15.170:0/3663213054 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268469967 0 0) 0x7f5b24006e90 con 0x7f5b240e2c40 2024-03-29T11:19:04.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.490 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.491 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:19:04.491 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 1 -- 172.21.15.170:0/3663213054 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b10005760 con 0x7f5b240e2c40 2024-03-29T11:19:04.491 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 1 -- 172.21.15.170:0/3663213054 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5b2c04f400 con 0x7f5b24029f00 2024-03-29T11:19:04.491 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.491 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:04.491 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 monclient(hunting): dump: 2024-03-29T11:19:04.491 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:04.491 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:04.491 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:04.491 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:04.491 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.491 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:04.491 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:04.491 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:04.491 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:04.491 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:04.492 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:04.492 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 1 -- 172.21.15.170:0/3663213054 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3817052292 0 0) 0x7f5b10005300 con 0x7f5b24029f00 2024-03-29T11:19:04.492 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 1 -- 172.21.15.170:0/3663213054 >> v1:172.21.15.184:6789/0 conn(0x7f5b2402b090 legacy=0x7f5b24037260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.492 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 1 -- 172.21.15.170:0/3663213054 >> v1:172.21.15.170:6789/0 conn(0x7f5b240e2c40 legacy=0x7f5b24184d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.492 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:04.492 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 1 monclient: found mon.c 2024-03-29T11:19:04.492 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:04.492 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:04.492 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 1 -- 172.21.15.170:0/3663213054 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240ddfa0 con 0x7f5b24029f00 2024-03-29T11:19:04.492 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:04.492 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:04.492 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.506+0000 7f5b227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:04.492 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.507+0000 7f5b32e8b640 5 monclient: authenticate success, global_id 4526 2024-03-29T11:19:04.492 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.507+0000 7f5b32e8b640 10 monclient: _renew_subs 2024-03-29T11:19:04.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.507+0000 7f5b32e8b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:04.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.507+0000 7f5b32e8b640 1 -- 172.21.15.170:0/3663213054 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b240de280 con 0x7f5b24029f00 2024-03-29T11:19:04.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.507+0000 7f5b32e8b640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:04.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.507+0000 7f5b32e8b640 10 monclient: _renew_subs 2024-03-29T11:19:04.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.507+0000 7f5b32e8b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:04.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.507+0000 7f5b32e8b640 1 -- 172.21.15.170:0/3663213054 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b24038970 con 0x7f5b24029f00 2024-03-29T11:19:04.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.507+0000 7f5b32e8b640 1 librados: init done 2024-03-29T11:19:04.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.507+0000 7f5b227fc640 1 -- 172.21.15.170:0/3663213054 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b2c04f730 con 0x7f5b24029f00 2024-03-29T11:19:04.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.507+0000 7f5b227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:04.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.507+0000 7f5b227fc640 1 -- 172.21.15.170:0/3663213054 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5b2c067d90 con 0x7f5b24029f00 2024-03-29T11:19:04.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.507+0000 7f5b227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.507+0000 7f5b227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:04.493 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.507+0000 7f5b227fc640 10 monclient: dump: 2024-03-29T11:19:04.493 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:04.493 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:04.494 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:04.494 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:04.494 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.494 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:04.494 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:04.494 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:04.494 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:04.494 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:04.494 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.507+0000 7f5b23fff640 4 set_mon_vals no callback set 2024-03-29T11:19:04.494 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.509+0000 7f5b227fc640 1 -- 172.21.15.170:0/3663213054 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f5b2c067ef0 con 0x7f5b24029f00 2024-03-29T11:19:04.494 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.510+0000 7f5b227fc640 1 -- 172.21.15.170:0/3663213054 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f5b2c0ee560 con 0x7f5b24029f00 2024-03-29T11:19:04.494 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.510+0000 7f5b227fc640 10 client.4526.objecter ms_dispatch 0x7f5b2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:04.494 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.510+0000 7f5b227fc640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:04.494 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.510+0000 7f5b227fc640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:04.494 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.510+0000 7f5b227fc640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-29T11:19:04.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.510+0000 7f5b32e8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:04.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.510+0000 7f5b32e8b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:04.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.510+0000 7f5b32e8b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:04.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.510+0000 7f5b32e8b640 1 -- 172.21.15.170:0/3663213054 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b14005080 con 0x7f5b24029f00 2024-03-29T11:19:04.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.511+0000 7f5b31688640 10 monclient: get_auth_request con 0x7f5b1006e070 auth_method 0 2024-03-29T11:19:04.495 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.511+0000 7f5b227fc640 10 client.4526.objecter ms_handle_connect 0x7f5b1006e070 2024-03-29T11:19:04.497 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.521+0000 7f5b227fc640 1 -- 172.21.15.170:0/3663213054 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b2c0b9ea0 con 0x7f5b24029f00 2024-03-29T11:19:04.497 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.521+0000 7f5b227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:04.498 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.521+0000 7f5b227fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:04.821 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.844+0000 7f5b32e8b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:04.821 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.844+0000 7f5b32e8b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:04.821 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.844+0000 7f5b32e8b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:04.821 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.844+0000 7f5b32e8b640 1 -- 172.21.15.170:0/3663213054 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b14005330 con 0x7f5b24029f00 2024-03-29T11:19:04.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.847+0000 7f5b227fc640 1 -- 172.21.15.170:0/3663213054 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f5b2c0b9ea0 con 0x7f5b24029f00 2024-03-29T11:19:04.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.847+0000 7f5b227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:04.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.847+0000 7f5b227fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:04.824 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:19:04.824 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:04.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.850+0000 7f5b32e8b640 10 librados: watch_flush enter 2024-03-29T11:19:04.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.854+0000 7f5b32e8b640 10 librados: watch_flush exit 2024-03-29T11:19:04.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.854+0000 7f5b32e8b640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-29T11:19:04.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.854+0000 7f5b32e8b640 10 client.4526.objecter successfully canceled tick 2024-03-29T11:19:04.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.854+0000 7f5b32e8b640 1 -- 172.21.15.170:0/3663213054 >> v1:172.21.15.170:6816/3298994444 conn(0x7f5b1006e070 legacy=0x7f5b10070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.854+0000 7f5b32e8b640 10 monclient: shutdown 2024-03-29T11:19:04.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.854+0000 7f5b32e8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:04.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.854+0000 7f5b32e8b640 1 -- 172.21.15.170:0/3663213054 >> v1:172.21.15.170:6790/0 conn(0x7f5b24029f00 legacy=0x7f5b24036b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.855+0000 7f5b32e8b640 1 -- 172.21.15.170:0/3663213054 shutdown_connections 2024-03-29T11:19:04.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.855+0000 7f5b32e8b640 1 -- 172.21.15.170:0/3663213054 >> 172.21.15.170:0/3663213054 conn(0x7f5b240058f0 msgr2=0x7f5b240e4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:04.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.855+0000 7f5b32e8b640 1 -- 172.21.15.170:0/3663213054 shutdown_connections 2024-03-29T11:19:04.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.855+0000 7f5b32e8b640 1 -- 172.21.15.170:0/3663213054 wait complete. 2024-03-29T11:19:04.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:04.855+0000 7f5b32e8b640 1 librados: shutdown 2024-03-29T11:19:04.849 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:11.657 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:11.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.836+0000 7fa30077b640 10 monclient: get_monmap_and_config 2024-03-29T11:19:11.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.836+0000 7fa30077b640 10 monclient: build_initial_monmap 2024-03-29T11:19:11.813 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.836+0000 7fa30077b640 10 monclient: monmap: 2024-03-29T11:19:11.813 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:19:11.813 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:11.813 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:19:11.837049+0000 2024-03-29T11:19:11.813 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:19:11.837049+0000 2024-03-29T11:19:11.813 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:11.813 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:11.813 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:19:11.813 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:19:11.813 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:19:11.814 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:11.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.837+0000 7fa30077b640 1 Processor -- start 2024-03-29T11:19:11.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.837+0000 7fa30077b640 1 -- start start 2024-03-29T11:19:11.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.837+0000 7fa30077b640 10 monclient: init 2024-03-29T11:19:11.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.838+0000 7fa30077b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:11.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.838+0000 7fa30077b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:11.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.838+0000 7fa30077b640 10 monclient(hunting): picked mon.noname-c con 0x7fa2f80b4330 addr v1:172.21.15.184:6789/0 2024-03-29T11:19:11.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.838+0000 7fa30077b640 10 monclient(hunting): picked mon.noname-a con 0x7fa2f81470b0 addr v1:172.21.15.170:6789/0 2024-03-29T11:19:11.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.839+0000 7fa30077b640 10 monclient(hunting): picked mon.noname-b con 0x7fa2f81439b0 addr v1:172.21.15.170:6790/0 2024-03-29T11:19:11.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.839+0000 7fa30077b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f806f2a0 con 0x7fa2f81470b0 2024-03-29T11:19:11.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.839+0000 7fa30077b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f8146b90 con 0x7fa2f80b4330 2024-03-29T11:19:11.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.839+0000 7fa30077b640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f80b2ca0 con 0x7fa2f81439b0 2024-03-29T11:19:11.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.839+0000 7fa30077b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:11.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.839+0000 7fa30077b640 10 monclient(hunting): authenticate will time out at 1333.150084s 2024-03-29T11:19:11.818 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.841+0000 7fa2fed18640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fa2f81439b0 0x7fa2f8151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:48907/0 (socket says 172.21.15.170:48907) 2024-03-29T11:19:11.819 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.841+0000 7fa2fed18640 1 -- 172.21.15.170:0/1184503703 learned_addr learned my addr 172.21.15.170:0/1184503703 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:19:11.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.845+0000 7fa2fd515640 1 -- 172.21.15.170:0/1184503703 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213193995 0 0) 0x7fa2f80b2ca0 con 0x7fa2f81439b0 2024-03-29T11:19:11.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.845+0000 7fa2fd515640 10 monclient(hunting): _negotiate 2024-03-29T11:19:11.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.845+0000 7fa2fd515640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:11.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.845+0000 7fa2fd515640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:11.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.845+0000 7fa2fd515640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:19:11.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.845+0000 7fa2fd515640 1 -- 172.21.15.170:0/1184503703 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e4003520 con 0x7fa2f81439b0 2024-03-29T11:19:11.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.845+0000 7fa2fd515640 1 -- 172.21.15.170:0/1184503703 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681091635 0 0) 0x7fa2f806f2a0 con 0x7fa2f81470b0 2024-03-29T11:19:11.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 10 monclient(hunting): _negotiate 2024-03-29T11:19:11.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:11.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:11.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:19:11.823 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 1 -- 172.21.15.170:0/1184503703 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f80b2ca0 con 0x7fa2f81470b0 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 1 -- 172.21.15.170:0/1184503703 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1589958070 0 0) 0x7fa2f80b2ca0 con 0x7fa2f81470b0 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 1 -- 172.21.15.170:0/1184503703 >> v1:172.21.15.170:6790/0 conn(0x7fa2f81439b0 legacy=0x7fa2f8151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 1 -- 172.21.15.170:0/1184503703 >> v1:172.21.15.184:6789/0 conn(0x7fa2f80b4330 legacy=0x7fa2f80b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 10 monclient: _finish_hunting 0 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 1 monclient: found mon.noname-a 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 1 -- 172.21.15.170:0/1184503703 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f806e0c0 con 0x7fa2f81470b0 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 10 monclient: _finish_auth 0 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 10 monclient: _check_auth_tickets 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 1 -- 172.21.15.170:0/1184503703 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa2e8001670 con 0x7fa2f81470b0 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 10 monclient: dump: 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa30077b640 5 monclient: authenticate success, global_id 4491 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa30077b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 1 -- 172.21.15.170:0/1184503703 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2e80017d0 con 0x7fa2f81470b0 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa30077b640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 1 -- 172.21.15.170:0/1184503703 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa2e8001c20 con 0x7fa2f81470b0 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:11.826 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa30077b640 4 set_mon_vals no callback set 2024-03-29T11:19:11.826 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:11.826 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.846+0000 7fa2fd515640 10 monclient: dump: 2024-03-29T11:19:11.826 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:11.826 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:11.826 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:11.826 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:11.826 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:11.826 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:11.826 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:11.826 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:11.826 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:11.826 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:11.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.847+0000 7fa30077b640 10 monclient: shutdown 2024-03-29T11:19:11.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.847+0000 7fa30077b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:11.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.847+0000 7fa30077b640 1 -- 172.21.15.170:0/1184503703 >> v1:172.21.15.170:6789/0 conn(0x7fa2f81470b0 legacy=0x7fa2f80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:11.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.847+0000 7fa30077b640 1 -- 172.21.15.170:0/1184503703 shutdown_connections 2024-03-29T11:19:11.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.847+0000 7fa30077b640 1 -- 172.21.15.170:0/1184503703 >> 172.21.15.170:0/1184503703 conn(0x7fa2f8066a00 msgr2=0x7fa2f8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:11.827 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.850+0000 7fa30077b640 1 -- 172.21.15.170:0/1184503703 shutdown_connections 2024-03-29T11:19:11.828 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.851+0000 7fa30077b640 1 -- 172.21.15.170:0/1184503703 wait complete. 2024-03-29T11:19:11.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.852+0000 7fa30077b640 10 monclient: build_initial_monmap 2024-03-29T11:19:11.830 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.852+0000 7fa30077b640 10 monclient: monmap: 2024-03-29T11:19:11.830 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:19:11.830 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:11.830 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:19:11.831 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:19:11.831 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:11.831 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:11.831 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:19:11.831 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:19:11.831 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:19:11.831 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:11.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.853+0000 7fa30077b640 1 librados: starting msgr at 2024-03-29T11:19:11.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.853+0000 7fa30077b640 1 librados: starting objecter 2024-03-29T11:19:11.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.853+0000 7fa30077b640 1 Processor -- start 2024-03-29T11:19:11.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.853+0000 7fa30077b640 1 -- start start 2024-03-29T11:19:11.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.853+0000 7fa30077b640 1 librados: setting wanted keys 2024-03-29T11:19:11.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.853+0000 7fa30077b640 1 librados: calling monclient init 2024-03-29T11:19:11.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.853+0000 7fa30077b640 10 monclient: init 2024-03-29T11:19:11.831 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.854+0000 7fa30077b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:11.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.854+0000 7fa30077b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:11.832 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.854+0000 7fa30077b640 10 monclient(hunting): picked mon.noname-c con 0x7fa2f80b4330 addr v1:172.21.15.170:6790/0 2024-03-29T11:19:11.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.858+0000 7fa30077b640 10 monclient(hunting): picked mon.noname-a con 0x7fa2f81439b0 addr v1:172.21.15.170:6789/0 2024-03-29T11:19:11.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.858+0000 7fa30077b640 10 monclient(hunting): picked mon.noname-b con 0x7fa2f81470b0 addr v1:172.21.15.184:6789/0 2024-03-29T11:19:11.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.858+0000 7fa2fdd16640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fa2f81439b0 0x7fa2f81d8890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:59969/0 (socket says 172.21.15.170:59969) 2024-03-29T11:19:11.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.858+0000 7fa2fdd16640 1 -- 172.21.15.170:0/3463661375 learned_addr learned my addr 172.21.15.170:0/3463661375 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:19:11.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.858+0000 7fa30077b640 1 -- 172.21.15.170:0/3463661375 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f8140ff0 con 0x7fa2f81439b0 2024-03-29T11:19:11.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.858+0000 7fa2eeffd640 10 client.?.objecter ms_handle_connect 0x7fa2f81439b0 2024-03-29T11:19:11.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.858+0000 7fa2eeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:11.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.858+0000 7fa30077b640 1 -- 172.21.15.170:0/3463661375 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f81db730 con 0x7fa2f81470b0 2024-03-29T11:19:11.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.858+0000 7fa30077b640 1 -- 172.21.15.170:0/3463661375 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f813f010 con 0x7fa2f80b4330 2024-03-29T11:19:11.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.858+0000 7fa30077b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:11.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.858+0000 7fa2eeffd640 1 -- 172.21.15.170:0/3463661375 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325512564 0 0) 0x7fa2f8140ff0 con 0x7fa2f81439b0 2024-03-29T11:19:11.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.858+0000 7fa30077b640 10 monclient(hunting): authenticate will time out at 1333.169903s 2024-03-29T11:19:11.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.859+0000 7fa2eeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:11.837 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.859+0000 7fa2eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:11.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.859+0000 7fa2eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:11.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.859+0000 7fa2eeffd640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:19:11.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.859+0000 7fa2eeffd640 1 -- 172.21.15.170:0/3463661375 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2dc004eb0 con 0x7fa2f81439b0 2024-03-29T11:19:11.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.859+0000 7fa2eeffd640 1 -- 172.21.15.170:0/3463661375 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3175437167 0 0) 0x7fa2dc004eb0 con 0x7fa2f81439b0 2024-03-29T11:19:11.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.859+0000 7fa2eeffd640 1 -- 172.21.15.170:0/3463661375 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2f8140ff0 con 0x7fa2f81439b0 2024-03-29T11:19:11.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.859+0000 7fa2eeffd640 1 -- 172.21.15.170:0/3463661375 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa2e8001630 con 0x7fa2f81439b0 2024-03-29T11:19:11.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.859+0000 7fa2eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:11.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.859+0000 7fa2eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:11.838 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.859+0000 7fa2eeffd640 10 monclient(hunting): dump: 2024-03-29T11:19:11.838 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:11.838 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:11.838 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:11.838 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:11.838 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:11.838 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:11.839 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:11.839 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:11.839 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:11.839 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:11.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.859+0000 7fa2eeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:11.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.860+0000 7fa2eeffd640 1 -- 172.21.15.170:0/3463661375 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2469734923 0 0) 0x7fa2f8140ff0 con 0x7fa2f81439b0 2024-03-29T11:19:11.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.860+0000 7fa2eeffd640 1 -- 172.21.15.170:0/3463661375 >> v1:172.21.15.170:6790/0 conn(0x7fa2f80b4330 legacy=0x7fa2f81dbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:11.839 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.860+0000 7fa2eeffd640 1 -- 172.21.15.170:0/3463661375 >> v1:172.21.15.184:6789/0 conn(0x7fa2f81470b0 legacy=0x7fa2f813f5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:11.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.862+0000 7fa2eeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:11.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.862+0000 7fa2eeffd640 1 monclient: found mon.a 2024-03-29T11:19:11.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.862+0000 7fa2eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:11.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.862+0000 7fa2eeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:19:11.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.862+0000 7fa2eeffd640 1 -- 172.21.15.170:0/3463661375 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f81d8250 con 0x7fa2f81439b0 2024-03-29T11:19:11.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.862+0000 7fa2eeffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:11.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.862+0000 7fa2eeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:11.842 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.862+0000 7fa2eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:11.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.863+0000 7fa30077b640 5 monclient: authenticate success, global_id 4494 2024-03-29T11:19:11.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.863+0000 7fa30077b640 10 monclient: _renew_subs 2024-03-29T11:19:11.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.863+0000 7fa30077b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:19:11.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.863+0000 7fa30077b640 1 -- 172.21.15.170:0/3463661375 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f81dc2d0 con 0x7fa2f81439b0 2024-03-29T11:19:11.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.863+0000 7fa30077b640 10 client.4494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:11.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.863+0000 7fa30077b640 10 monclient: _renew_subs 2024-03-29T11:19:11.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.863+0000 7fa30077b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:19:11.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.863+0000 7fa30077b640 1 -- 172.21.15.170:0/3463661375 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f81d9180 con 0x7fa2f81439b0 2024-03-29T11:19:11.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.863+0000 7fa30077b640 1 librados: init done 2024-03-29T11:19:11.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.865+0000 7fa2eeffd640 1 -- 172.21.15.170:0/3463661375 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2e80038a0 con 0x7fa2f81439b0 2024-03-29T11:19:11.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.865+0000 7fa2eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:11.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.865+0000 7fa2eeffd640 1 -- 172.21.15.170:0/3463661375 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fa2e8003c70 con 0x7fa2f81439b0 2024-03-29T11:19:11.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.865+0000 7fa2eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:11.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.865+0000 7fa2eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:11.843 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.865+0000 7fa2eeffd640 10 monclient: dump: 2024-03-29T11:19:11.844 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:11.844 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:11.844 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:11.844 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:11.844 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:11.844 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:11.844 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:11.844 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:11.844 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:11.844 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:11.844 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.865+0000 7fa30077b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:11.844 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.865+0000 7fa30077b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:11.844 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.865+0000 7fa30077b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:19:11.844 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.865+0000 7fa30077b640 1 -- 172.21.15.170:0/3463661375 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f81d8440 con 0x7fa2f81439b0 2024-03-29T11:19:11.844 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.865+0000 7fa2fcd14640 4 set_mon_vals no callback set 2024-03-29T11:19:11.847 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.868+0000 7fa2eeffd640 1 -- 172.21.15.170:0/3463661375 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fa2e8003dd0 con 0x7fa2f81439b0 2024-03-29T11:19:11.847 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.869+0000 7fa2fe517640 10 monclient: get_auth_request con 0x7fa2dc06dc80 auth_method 0 2024-03-29T11:19:11.847 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.870+0000 7fa2eeffd640 1 -- 172.21.15.170:0/3463661375 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fa2e8084b70 con 0x7fa2f81439b0 2024-03-29T11:19:11.847 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.870+0000 7fa2eeffd640 10 client.4494.objecter ms_dispatch 0x7fa2f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:11.847 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.870+0000 7fa2eeffd640 3 client.4494.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:11.847 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.870+0000 7fa2eeffd640 3 client.4494.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:11.847 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.870+0000 7fa2eeffd640 20 client.4494.objecter dump_active .. 0 homeless 2024-03-29T11:19:11.847 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.870+0000 7fa2eeffd640 10 client.4494.objecter ms_handle_connect 0x7fa2dc06dc80 2024-03-29T11:19:11.852 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.875+0000 7fa2eeffd640 1 -- 172.21.15.170:0/3463661375 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa2e8050450 con 0x7fa2f81439b0 2024-03-29T11:19:11.852 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.875+0000 7fa2eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:11.852 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:11.875+0000 7fa2eeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:12.179 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.202+0000 7fa30077b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:12.179 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.202+0000 7fa30077b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:12.179 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.202+0000 7fa30077b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:19:12.179 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.202+0000 7fa30077b640 1 -- 172.21.15.170:0/3463661375 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2f8075f10 con 0x7fa2f81439b0 2024-03-29T11:19:12.181 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.204+0000 7fa2eeffd640 1 -- 172.21.15.170:0/3463661375 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fa2e8052380 con 0x7fa2f81439b0 2024-03-29T11:19:12.181 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.204+0000 7fa2eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:12.181 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.204+0000 7fa2eeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:12.182 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:19:12.182 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:12.186 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.208+0000 7fa30077b640 10 librados: watch_flush enter 2024-03-29T11:19:12.186 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.209+0000 7fa30077b640 10 librados: watch_flush exit 2024-03-29T11:19:12.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.209+0000 7fa30077b640 20 client.4494.objecter shutdown clearing up homeless session... 2024-03-29T11:19:12.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.209+0000 7fa30077b640 10 client.4494.objecter successfully canceled tick 2024-03-29T11:19:12.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.210+0000 7fa30077b640 1 -- 172.21.15.170:0/3463661375 >> v1:172.21.15.170:6816/3298994444 conn(0x7fa2dc06dc80 legacy=0x7fa2dc070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:12.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.210+0000 7fa30077b640 10 monclient: shutdown 2024-03-29T11:19:12.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.210+0000 7fa30077b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:12.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.210+0000 7fa30077b640 1 -- 172.21.15.170:0/3463661375 >> v1:172.21.15.170:6789/0 conn(0x7fa2f81439b0 legacy=0x7fa2f81d8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:12.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.210+0000 7fa30077b640 1 -- 172.21.15.170:0/3463661375 shutdown_connections 2024-03-29T11:19:12.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.210+0000 7fa30077b640 1 -- 172.21.15.170:0/3463661375 >> 172.21.15.170:0/3463661375 conn(0x7fa2f8066a00 msgr2=0x7fa2f8145aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:12.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.210+0000 7fa30077b640 1 -- 172.21.15.170:0/3463661375 shutdown_connections 2024-03-29T11:19:12.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.210+0000 7fa30077b640 1 -- 172.21.15.170:0/3463661375 wait complete. 2024-03-29T11:19:12.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:12.210+0000 7fa30077b640 1 librados: shutdown 2024-03-29T11:19:12.207 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:19.014 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:19.166 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.188+0000 7fd388e14640 10 monclient: get_monmap_and_config 2024-03-29T11:19:19.166 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.188+0000 7fd388e14640 10 monclient: build_initial_monmap 2024-03-29T11:19:19.166 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.188+0000 7fd388e14640 10 monclient: monmap: 2024-03-29T11:19:19.166 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:19:19.166 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:19.166 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:19:19.188998+0000 2024-03-29T11:19:19.166 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:19:19.188998+0000 2024-03-29T11:19:19.166 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:19.166 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:19.166 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:19:19.166 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:19:19.167 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:19:19.167 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:19.169 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.191+0000 7fd388e14640 1 Processor -- start 2024-03-29T11:19:19.169 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.191+0000 7fd388e14640 1 -- start start 2024-03-29T11:19:19.169 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.191+0000 7fd388e14640 10 monclient: init 2024-03-29T11:19:19.169 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.192+0000 7fd388e14640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:19.169 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.192+0000 7fd388e14640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:19.169 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.192+0000 7fd388e14640 10 monclient(hunting): picked mon.noname-c con 0x7fd3840b01a0 addr v1:172.21.15.184:6789/0 2024-03-29T11:19:19.169 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.192+0000 7fd388e14640 10 monclient(hunting): picked mon.noname-b con 0x7fd3840af010 addr v1:172.21.15.170:6790/0 2024-03-29T11:19:19.169 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.192+0000 7fd388e14640 10 monclient(hunting): picked mon.noname-a con 0x7fd3840ac5a0 addr v1:172.21.15.170:6789/0 2024-03-29T11:19:19.169 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.192+0000 7fd388e14640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3840ae8a0 con 0x7fd3840ac5a0 2024-03-29T11:19:19.169 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.192+0000 7fd388e14640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3840ab1d0 con 0x7fd3840b01a0 2024-03-29T11:19:19.169 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.192+0000 7fd388e14640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3840ab4b0 con 0x7fd3840af010 2024-03-29T11:19:19.169 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.192+0000 7fd388e14640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:19.169 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.192+0000 7fd388e14640 10 monclient(hunting): authenticate will time out at 1340.503314s 2024-03-29T11:19:19.173 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.197+0000 7fd38259c640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fd3840b01a0 0x7fd3840abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:59927/0 (socket says 172.21.15.170:59927) 2024-03-29T11:19:19.173 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.197+0000 7fd38259c640 1 -- 172.21.15.170:0/2482416926 learned_addr learned my addr 172.21.15.170:0/2482416926 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:19:19.175 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.197+0000 7fd38159a640 1 -- 172.21.15.170:0/2482416926 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546287016 0 0) 0x7fd3840ae8a0 con 0x7fd3840ac5a0 2024-03-29T11:19:19.175 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.197+0000 7fd38159a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:19.175 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.197+0000 7fd38159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:19.175 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.197+0000 7fd38159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:19.175 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.197+0000 7fd38159a640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:19:19.175 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.197+0000 7fd38159a640 1 -- 172.21.15.170:0/2482416926 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd36c003520 con 0x7fd3840ac5a0 2024-03-29T11:19:19.175 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.197+0000 7fd38159a640 1 -- 172.21.15.170:0/2482416926 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609395030 0 0) 0x7fd3840ab1d0 con 0x7fd3840b01a0 2024-03-29T11:19:19.175 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.197+0000 7fd38159a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:19.175 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.197+0000 7fd38159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:19.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.197+0000 7fd38159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:19.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.197+0000 7fd38159a640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:19:19.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.197+0000 7fd38159a640 1 -- 172.21.15.170:0/2482416926 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3840ae8a0 con 0x7fd3840b01a0 2024-03-29T11:19:19.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.198+0000 7fd38159a640 1 -- 172.21.15.170:0/2482416926 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4225161423 0 0) 0x7fd36c003520 con 0x7fd3840ac5a0 2024-03-29T11:19:19.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.198+0000 7fd38159a640 1 -- 172.21.15.170:0/2482416926 >> v1:172.21.15.170:6790/0 conn(0x7fd3840af010 legacy=0x7fd384149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:19.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.198+0000 7fd38159a640 1 -- 172.21.15.170:0/2482416926 >> v1:172.21.15.184:6789/0 conn(0x7fd3840b01a0 legacy=0x7fd3840abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:19.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.198+0000 7fd38159a640 10 monclient: _finish_hunting 0 2024-03-29T11:19:19.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.198+0000 7fd38159a640 1 monclient: found mon.noname-a 2024-03-29T11:19:19.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.198+0000 7fd38159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:19.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.198+0000 7fd38159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:19:19.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.198+0000 7fd38159a640 1 -- 172.21.15.170:0/2482416926 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3840ab790 con 0x7fd3840ac5a0 2024-03-29T11:19:19.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.198+0000 7fd38159a640 10 monclient: _finish_auth 0 2024-03-29T11:19:19.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.198+0000 7fd38159a640 10 monclient: _check_auth_tickets 2024-03-29T11:19:19.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.198+0000 7fd38159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:19.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.198+0000 7fd38159a640 1 -- 172.21.15.170:0/2482416926 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fd3780015c0 con 0x7fd3840ac5a0 2024-03-29T11:19:19.177 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.198+0000 7fd38159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:19.177 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.198+0000 7fd38159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:19.177 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.198+0000 7fd38159a640 10 monclient: dump: 2024-03-29T11:19:19.177 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:19.177 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:19.177 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:19.177 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:19.177 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:19.177 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:19.177 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:19.177 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:19.177 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:19.177 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:19.178 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.201+0000 7fd388e14640 5 monclient: authenticate success, global_id 4497 2024-03-29T11:19:19.178 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.201+0000 7fd388e14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:19.179 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.202+0000 7fd38159a640 1 -- 172.21.15.170:0/2482416926 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3780017e0 con 0x7fd3840ac5a0 2024-03-29T11:19:19.179 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.202+0000 7fd38159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:19.179 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.202+0000 7fd38159a640 1 -- 172.21.15.170:0/2482416926 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fd378001b70 con 0x7fd3840ac5a0 2024-03-29T11:19:19.179 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.202+0000 7fd38159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:19.179 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.202+0000 7fd38159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:19.179 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.202+0000 7fd38159a640 10 monclient: dump: 2024-03-29T11:19:19.179 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:19.179 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:19.179 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:19.179 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:19.179 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:19.179 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:19.179 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:19.179 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:19.180 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:19.180 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:19.180 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.202+0000 7fd388e14640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:19.180 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.202+0000 7fd388e14640 4 set_mon_vals no callback set 2024-03-29T11:19:19.180 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.203+0000 7fd388e14640 10 monclient: shutdown 2024-03-29T11:19:19.180 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.203+0000 7fd388e14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:19.180 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.203+0000 7fd388e14640 1 -- 172.21.15.170:0/2482416926 >> v1:172.21.15.170:6789/0 conn(0x7fd3840ac5a0 legacy=0x7fd384151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:19.182 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.205+0000 7fd388e14640 1 -- 172.21.15.170:0/2482416926 shutdown_connections 2024-03-29T11:19:19.182 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.205+0000 7fd388e14640 1 -- 172.21.15.170:0/2482416926 >> 172.21.15.170:0/2482416926 conn(0x7fd38406b660 msgr2=0x7fd38406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:19.182 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.205+0000 7fd388e14640 1 -- 172.21.15.170:0/2482416926 shutdown_connections 2024-03-29T11:19:19.182 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.205+0000 7fd388e14640 1 -- 172.21.15.170:0/2482416926 wait complete. 2024-03-29T11:19:19.184 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.207+0000 7fd388e14640 10 monclient: build_initial_monmap 2024-03-29T11:19:19.184 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.207+0000 7fd388e14640 10 monclient: monmap: 2024-03-29T11:19:19.184 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:19:19.184 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:19.184 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:19:19.184 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:19:19.184 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:19.184 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:19.184 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:19:19.185 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:19:19.185 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:19:19.185 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:19.185 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.207+0000 7fd388e14640 1 librados: starting msgr at 2024-03-29T11:19:19.185 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.207+0000 7fd388e14640 1 librados: starting objecter 2024-03-29T11:19:19.185 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.207+0000 7fd388e14640 1 Processor -- start 2024-03-29T11:19:19.186 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.208+0000 7fd388e14640 1 -- start start 2024-03-29T11:19:19.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.208+0000 7fd388e14640 1 librados: setting wanted keys 2024-03-29T11:19:19.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.208+0000 7fd388e14640 1 librados: calling monclient init 2024-03-29T11:19:19.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.208+0000 7fd388e14640 10 monclient: init 2024-03-29T11:19:19.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.208+0000 7fd388e14640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:19.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.208+0000 7fd388e14640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:19.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.208+0000 7fd388e14640 10 monclient(hunting): picked mon.noname-c con 0x7fd3840ac5a0 addr v1:172.21.15.170:6790/0 2024-03-29T11:19:19.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.208+0000 7fd388e14640 10 monclient(hunting): picked mon.noname-a con 0x7fd3840b01a0 addr v1:172.21.15.170:6789/0 2024-03-29T11:19:19.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.208+0000 7fd388e14640 10 monclient(hunting): picked mon.noname-b con 0x7fd3841250e0 addr v1:172.21.15.184:6789/0 2024-03-29T11:19:19.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.208+0000 7fd388e14640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3841272f0 con 0x7fd3840b01a0 2024-03-29T11:19:19.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.208+0000 7fd388e14640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd384125bd0 con 0x7fd3841250e0 2024-03-29T11:19:19.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.208+0000 7fd388e14640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3841260f0 con 0x7fd3840ac5a0 2024-03-29T11:19:19.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.208+0000 7fd388e14640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:19.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.208+0000 7fd388e14640 10 monclient(hunting): authenticate will time out at 1340.519860s 2024-03-29T11:19:19.193 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.215+0000 7fd381d9b640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fd3840b01a0 0x7fd384143f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:37361/0 (socket says 172.21.15.170:37361) 2024-03-29T11:19:19.193 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.215+0000 7fd381d9b640 1 -- 172.21.15.170:0/1463345426 learned_addr learned my addr 172.21.15.170:0/1463345426 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:19:19.193 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.215+0000 7fd372ffd640 10 client.?.objecter ms_handle_connect 0x7fd3840b01a0 2024-03-29T11:19:19.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.215+0000 7fd372ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:19.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.215+0000 7fd372ffd640 1 -- 172.21.15.170:0/1463345426 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597101209 0 0) 0x7fd3841272f0 con 0x7fd3840b01a0 2024-03-29T11:19:19.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.215+0000 7fd372ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:19.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.215+0000 7fd372ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:19.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.215+0000 7fd372ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:19.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.215+0000 7fd372ffd640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:19:19.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.215+0000 7fd372ffd640 1 -- 172.21.15.170:0/1463345426 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd364004eb0 con 0x7fd3840b01a0 2024-03-29T11:19:19.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.216+0000 7fd372ffd640 1 -- 172.21.15.170:0/1463345426 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2096855088 0 0) 0x7fd364004eb0 con 0x7fd3840b01a0 2024-03-29T11:19:19.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.216+0000 7fd372ffd640 1 -- 172.21.15.170:0/1463345426 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3841272f0 con 0x7fd3840b01a0 2024-03-29T11:19:19.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.216+0000 7fd372ffd640 10 client.?.objecter ms_handle_connect 0x7fd3840ac5a0 2024-03-29T11:19:19.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.216+0000 7fd372ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:19.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.216+0000 7fd372ffd640 1 -- 172.21.15.170:0/1463345426 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fd3740019d0 con 0x7fd3840b01a0 2024-03-29T11:19:19.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.216+0000 7fd372ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:19.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.216+0000 7fd372ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:19.195 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.216+0000 7fd372ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:19.195 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:19.195 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:19.195 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:19.195 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:19.195 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:19.195 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:19.195 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:19.195 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:19.195 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:19.195 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:19.195 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.216+0000 7fd372ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:19.195 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.216+0000 7fd372ffd640 1 -- 172.21.15.170:0/1463345426 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3772538987 0 0) 0x7fd3841272f0 con 0x7fd3840b01a0 2024-03-29T11:19:19.195 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.216+0000 7fd372ffd640 1 -- 172.21.15.170:0/1463345426 >> v1:172.21.15.170:6790/0 conn(0x7fd3840ac5a0 legacy=0x7fd3841db590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:19.195 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.216+0000 7fd372ffd640 1 -- 172.21.15.170:0/1463345426 >> v1:172.21.15.184:6789/0 conn(0x7fd3841250e0 legacy=0x7fd3841e6eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:19.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.218+0000 7fd372ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:19.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.218+0000 7fd372ffd640 1 monclient: found mon.a 2024-03-29T11:19:19.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.218+0000 7fd372ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:19.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.218+0000 7fd372ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:19:19.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.218+0000 7fd372ffd640 1 -- 172.21.15.170:0/1463345426 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3841dbca0 con 0x7fd3840b01a0 2024-03-29T11:19:19.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.218+0000 7fd372ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:19.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.218+0000 7fd372ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:19.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.218+0000 7fd372ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:19.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.219+0000 7fd372ffd640 1 -- 172.21.15.170:0/1463345426 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd374001cd0 con 0x7fd3840b01a0 2024-03-29T11:19:19.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.219+0000 7fd372ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:19.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.219+0000 7fd380d99640 4 set_mon_vals no callback set 2024-03-29T11:19:19.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.219+0000 7fd372ffd640 1 -- 172.21.15.170:0/1463345426 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fd374003a60 con 0x7fd3840b01a0 2024-03-29T11:19:19.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.219+0000 7fd372ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:19.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.220+0000 7fd372ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:19.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.220+0000 7fd372ffd640 10 monclient: dump: 2024-03-29T11:19:19.199 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:19.200 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:19.200 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:19.200 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:19.200 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:19.200 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:19.200 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:19.200 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:19.200 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:19.200 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:19.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.221+0000 7fd388e14640 5 monclient: authenticate success, global_id 4500 2024-03-29T11:19:19.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.221+0000 7fd388e14640 10 monclient: _renew_subs 2024-03-29T11:19:19.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.221+0000 7fd388e14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:19:19.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.221+0000 7fd388e14640 1 -- 172.21.15.170:0/1463345426 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3841dbef0 con 0x7fd3840b01a0 2024-03-29T11:19:19.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.221+0000 7fd388e14640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:19.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.221+0000 7fd388e14640 10 monclient: _renew_subs 2024-03-29T11:19:19.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.221+0000 7fd388e14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:19:19.201 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.221+0000 7fd388e14640 1 -- 172.21.15.170:0/1463345426 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd384144810 con 0x7fd3840b01a0 2024-03-29T11:19:19.201 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.221+0000 7fd388e14640 1 librados: init done 2024-03-29T11:19:19.201 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.223+0000 7fd370ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:19.201 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.223+0000 7fd370ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:19.201 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.223+0000 7fd370ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:19:19.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.223+0000 7fd370ff9640 1 -- 172.21.15.170:0/1463345426 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd384125d60 con 0x7fd3840b01a0 2024-03-29T11:19:19.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.226+0000 7fd372ffd640 1 -- 172.21.15.170:0/1463345426 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fd374003c40 con 0x7fd3840b01a0 2024-03-29T11:19:19.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.226+0000 7fd372ffd640 1 -- 172.21.15.170:0/1463345426 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fd3740849e0 con 0x7fd3840b01a0 2024-03-29T11:19:19.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.226+0000 7fd372ffd640 10 client.4500.objecter ms_dispatch 0x7fd384000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:19.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.226+0000 7fd372ffd640 3 client.4500.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:19.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.226+0000 7fd372ffd640 3 client.4500.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:19.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.226+0000 7fd372ffd640 20 client.4500.objecter dump_active .. 0 homeless 2024-03-29T11:19:19.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.227+0000 7fd38259c640 10 monclient: get_auth_request con 0x7fd36406ded0 auth_method 0 2024-03-29T11:19:19.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.227+0000 7fd372ffd640 10 client.4500.objecter ms_handle_connect 0x7fd36406ded0 2024-03-29T11:19:19.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.232+0000 7fd372ffd640 1 -- 172.21.15.170:0/1463345426 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd3740500f0 con 0x7fd3840b01a0 2024-03-29T11:19:19.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.232+0000 7fd372ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:19.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.232+0000 7fd372ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:19.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.571+0000 7fd370ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:19.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.571+0000 7fd370ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:19.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.571+0000 7fd370ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:19:19.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.571+0000 7fd370ff9640 1 -- 172.21.15.170:0/1463345426 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd384075f10 con 0x7fd3840b01a0 2024-03-29T11:19:19.549 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.573+0000 7fd372ffd640 1 -- 172.21.15.170:0/1463345426 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fd3740038c0 con 0x7fd3840b01a0 2024-03-29T11:19:19.549 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.573+0000 7fd372ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:19.549 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.573+0000 7fd372ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:19.553 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:19:19.553 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:19.554 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.577+0000 7fd370ff9640 10 librados: watch_flush enter 2024-03-29T11:19:19.554 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.577+0000 7fd370ff9640 10 librados: watch_flush exit 2024-03-29T11:19:19.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.577+0000 7fd370ff9640 20 client.4500.objecter shutdown clearing up homeless session... 2024-03-29T11:19:19.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.577+0000 7fd370ff9640 10 client.4500.objecter successfully canceled tick 2024-03-29T11:19:19.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.577+0000 7fd370ff9640 1 -- 172.21.15.170:0/1463345426 >> v1:172.21.15.170:6816/3298994444 conn(0x7fd36406ded0 legacy=0x7fd364070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:19.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.577+0000 7fd370ff9640 10 monclient: shutdown 2024-03-29T11:19:19.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.577+0000 7fd370ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:19.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.577+0000 7fd370ff9640 1 -- 172.21.15.170:0/1463345426 >> v1:172.21.15.170:6789/0 conn(0x7fd3840b01a0 legacy=0x7fd384143f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:19.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.581+0000 7fd370ff9640 1 -- 172.21.15.170:0/1463345426 shutdown_connections 2024-03-29T11:19:19.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.581+0000 7fd370ff9640 1 -- 172.21.15.170:0/1463345426 >> 172.21.15.170:0/1463345426 conn(0x7fd38406b660 msgr2=0x7fd384149530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:19.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.588+0000 7fd370ff9640 1 -- 172.21.15.170:0/1463345426 shutdown_connections 2024-03-29T11:19:19.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.589+0000 7fd370ff9640 1 -- 172.21.15.170:0/1463345426 wait complete. 2024-03-29T11:19:19.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:19.589+0000 7fd370ff9640 1 librados: shutdown 2024-03-29T11:19:19.584 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:26.391 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:26.539 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.561+0000 7fb4af67b640 10 monclient: get_monmap_and_config 2024-03-29T11:19:26.539 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.561+0000 7fb4af67b640 10 monclient: build_initial_monmap 2024-03-29T11:19:26.539 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.562+0000 7fb4af67b640 10 monclient: monmap: 2024-03-29T11:19:26.539 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:19:26.539 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:26.539 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:19:26.563422+0000 2024-03-29T11:19:26.539 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:19:26.563422+0000 2024-03-29T11:19:26.540 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:26.540 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:26.540 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:19:26.540 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:19:26.540 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:19:26.540 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:26.541 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.564+0000 7fb4af67b640 1 Processor -- start 2024-03-29T11:19:26.541 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.564+0000 7fb4af67b640 1 -- start start 2024-03-29T11:19:26.541 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.564+0000 7fb4af67b640 10 monclient: init 2024-03-29T11:19:26.545 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.565+0000 7fb4af67b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:26.545 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.565+0000 7fb4af67b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:26.545 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.565+0000 7fb4af67b640 10 monclient(hunting): picked mon.noname-c con 0x7fb4a81457f0 addr v1:172.21.15.184:6789/0 2024-03-29T11:19:26.545 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.565+0000 7fb4af67b640 10 monclient(hunting): picked mon.noname-b con 0x7fb4a813f650 addr v1:172.21.15.170:6790/0 2024-03-29T11:19:26.545 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.565+0000 7fb4af67b640 10 monclient(hunting): picked mon.noname-a con 0x7fb4a80a9d70 addr v1:172.21.15.170:6789/0 2024-03-29T11:19:26.545 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.565+0000 7fb4af67b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a80aaef0 con 0x7fb4a80a9d70 2024-03-29T11:19:26.545 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.565+0000 7fb4af67b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a8145bd0 con 0x7fb4a81457f0 2024-03-29T11:19:26.545 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.565+0000 7fb4af67b640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a8145eb0 con 0x7fb4a813f650 2024-03-29T11:19:26.545 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.565+0000 7fb4af67b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:26.545 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.565+0000 7fb4af67b640 10 monclient(hunting): authenticate will time out at 1347.877254s 2024-03-29T11:19:26.545 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.567+0000 7fb4adc18640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fb4a80a9d70 0x7fb4a814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:49890/0 (socket says 172.21.15.170:49890) 2024-03-29T11:19:26.545 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.567+0000 7fb4adc18640 1 -- 172.21.15.170:0/2946528881 learned_addr learned my addr 172.21.15.170:0/2946528881 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:19:26.546 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.568+0000 7fb49ffff640 1 -- 172.21.15.170:0/2946528881 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588489870 0 0) 0x7fb4a80aaef0 con 0x7fb4a80a9d70 2024-03-29T11:19:26.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.568+0000 7fb49ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.568+0000 7fb49ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.568+0000 7fb49ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.568+0000 7fb49ffff640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:19:26.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.568+0000 7fb49ffff640 1 -- 172.21.15.170:0/2946528881 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb494003520 con 0x7fb4a80a9d70 2024-03-29T11:19:26.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.568+0000 7fb49ffff640 1 -- 172.21.15.170:0/2946528881 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168732482 0 0) 0x7fb4a8145eb0 con 0x7fb4a813f650 2024-03-29T11:19:26.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.568+0000 7fb49ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.568+0000 7fb49ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.568+0000 7fb49ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.568+0000 7fb49ffff640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:19:26.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.568+0000 7fb49ffff640 1 -- 172.21.15.170:0/2946528881 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4a80aaef0 con 0x7fb4a813f650 2024-03-29T11:19:26.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 1 -- 172.21.15.170:0/2946528881 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1577704720 0 0) 0x7fb494003520 con 0x7fb4a80a9d70 2024-03-29T11:19:26.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 1 -- 172.21.15.170:0/2946528881 >> v1:172.21.15.170:6790/0 conn(0x7fb4a813f650 legacy=0x7fb4a80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 1 -- 172.21.15.170:0/2946528881 >> v1:172.21.15.184:6789/0 conn(0x7fb4a81457f0 legacy=0x7fb4a813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:26.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:26.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 1 monclient: found mon.noname-a 2024-03-29T11:19:26.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:26.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:19:26.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 1 -- 172.21.15.170:0/2946528881 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a8146340 con 0x7fb4a80a9d70 2024-03-29T11:19:26.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 10 monclient: _finish_auth 0 2024-03-29T11:19:26.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:26.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:26.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 1 -- 172.21.15.170:0/2946528881 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb4a0001a10 con 0x7fb4a80a9d70 2024-03-29T11:19:26.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:26.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 10 monclient: dump: 2024-03-29T11:19:26.549 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:26.549 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:26.549 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:26.549 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:26.549 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.549 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:26.549 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:26.549 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:26.549 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:26.549 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:26.549 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 1 -- 172.21.15.170:0/2946528881 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4a0001be0 con 0x7fb4a80a9d70 2024-03-29T11:19:26.549 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:26.549 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 1 -- 172.21.15.170:0/2946528881 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb4a0001f70 con 0x7fb4a80a9d70 2024-03-29T11:19:26.549 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.549 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:26.550 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb49ffff640 10 monclient: dump: 2024-03-29T11:19:26.550 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:26.550 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:26.550 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:26.550 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:26.550 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.550 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:26.550 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:26.550 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:26.550 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:26.550 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:26.551 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb4af67b640 5 monclient: authenticate success, global_id 4503 2024-03-29T11:19:26.551 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb4af67b640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:26.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.569+0000 7fb4af67b640 4 set_mon_vals no callback set 2024-03-29T11:19:26.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.570+0000 7fb4af67b640 10 monclient: shutdown 2024-03-29T11:19:26.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.570+0000 7fb4af67b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:26.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.570+0000 7fb4af67b640 1 -- 172.21.15.170:0/2946528881 >> v1:172.21.15.170:6789/0 conn(0x7fb4a80a9d70 legacy=0x7fb4a814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.570+0000 7fb4af67b640 1 -- 172.21.15.170:0/2946528881 shutdown_connections 2024-03-29T11:19:26.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.570+0000 7fb4af67b640 1 -- 172.21.15.170:0/2946528881 >> 172.21.15.170:0/2946528881 conn(0x7fb4a8066a00 msgr2=0x7fb4a8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:26.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.572+0000 7fb4af67b640 1 -- 172.21.15.170:0/2946528881 shutdown_connections 2024-03-29T11:19:26.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.572+0000 7fb4af67b640 1 -- 172.21.15.170:0/2946528881 wait complete. 2024-03-29T11:19:26.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.572+0000 7fb4af67b640 10 monclient: build_initial_monmap 2024-03-29T11:19:26.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.572+0000 7fb4af67b640 10 monclient: monmap: 2024-03-29T11:19:26.556 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:19:26.557 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:26.557 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:19:26.557 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:19:26.557 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:26.557 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:26.557 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:19:26.557 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:19:26.557 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:19:26.557 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:26.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.573+0000 7fb4af67b640 1 librados: starting msgr at 2024-03-29T11:19:26.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.573+0000 7fb4af67b640 1 librados: starting objecter 2024-03-29T11:19:26.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.573+0000 7fb4af67b640 1 Processor -- start 2024-03-29T11:19:26.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.573+0000 7fb4af67b640 1 -- start start 2024-03-29T11:19:26.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.573+0000 7fb4af67b640 1 librados: setting wanted keys 2024-03-29T11:19:26.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.573+0000 7fb4af67b640 1 librados: calling monclient init 2024-03-29T11:19:26.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.573+0000 7fb4af67b640 10 monclient: init 2024-03-29T11:19:26.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.574+0000 7fb4af67b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:26.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.574+0000 7fb4af67b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:26.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.574+0000 7fb4af67b640 10 monclient(hunting): picked mon.noname-a con 0x7fb4a80a9d70 addr v1:172.21.15.170:6789/0 2024-03-29T11:19:26.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.577+0000 7fb4af67b640 10 monclient(hunting): picked mon.noname-b con 0x7fb4a813f650 addr v1:172.21.15.184:6789/0 2024-03-29T11:19:26.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.577+0000 7fb4af67b640 10 monclient(hunting): picked mon.noname-c con 0x7fb4a81457f0 addr v1:172.21.15.170:6790/0 2024-03-29T11:19:26.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.577+0000 7fb4af67b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a8070c30 con 0x7fb4a80a9d70 2024-03-29T11:19:26.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.577+0000 7fb4af67b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a8070ea0 con 0x7fb4a813f650 2024-03-29T11:19:26.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.577+0000 7fb4af67b640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a8130310 con 0x7fb4a81457f0 2024-03-29T11:19:26.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.577+0000 7fb4ad417640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fb4a80a9d70 0x7fb4a8071910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:57556/0 (socket says 172.21.15.170:57556) 2024-03-29T11:19:26.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.577+0000 7fb4ad417640 1 -- 172.21.15.170:0/372195807 learned_addr learned my addr 172.21.15.170:0/372195807 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:19:26.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.577+0000 7fb49dffb640 10 client.?.objecter ms_handle_connect 0x7fb4a80a9d70 2024-03-29T11:19:26.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.577+0000 7fb49dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.577+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256642248 0 0) 0x7fb4a8070c30 con 0x7fb4a80a9d70 2024-03-29T11:19:26.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.577+0000 7fb4af67b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:26.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.578+0000 7fb4af67b640 10 monclient(hunting): authenticate will time out at 1347.890094s 2024-03-29T11:19:26.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.578+0000 7fb49dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.578+0000 7fb49dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.578+0000 7fb49dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.578+0000 7fb49dffb640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:19:26.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.578+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48c004eb0 con 0x7fb4a80a9d70 2024-03-29T11:19:26.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.578+0000 7fb49dffb640 10 client.?.objecter ms_handle_connect 0x7fb4a813f650 2024-03-29T11:19:26.559 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.578+0000 7fb49dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.561 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 10 client.?.objecter ms_handle_connect 0x7fb4a81457f0 2024-03-29T11:19:26.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 775712529 0 0) 0x7fb48c004eb0 con 0x7fb4a80a9d70 2024-03-29T11:19:26.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4a8070c30 con 0x7fb4a80a9d70 2024-03-29T11:19:26.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685523088 0 0) 0x7fb4a8070ea0 con 0x7fb4a813f650 2024-03-29T11:19:26.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:19:26.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48c004eb0 con 0x7fb4a813f650 2024-03-29T11:19:26.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172092118 0 0) 0x7fb4a8130310 con 0x7fb4a81457f0 2024-03-29T11:19:26.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:19:26.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4a8070ea0 con 0x7fb4a81457f0 2024-03-29T11:19:26.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb4980017d0 con 0x7fb4a80a9d70 2024-03-29T11:19:26.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:26.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 10 monclient(hunting): dump: 2024-03-29T11:19:26.563 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:26.563 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:26.563 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:26.563 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:26.563 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.563 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:26.563 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:26.563 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:26.564 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:26.564 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:26.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.581+0000 7fb49dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:26.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1555832983 0 0) 0x7fb48c004eb0 con 0x7fb4a813f650 2024-03-29T11:19:26.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4a8130310 con 0x7fb4a813f650 2024-03-29T11:19:26.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb4a40016b0 con 0x7fb4a813f650 2024-03-29T11:19:26.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:26.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 10 monclient(hunting): dump: 2024-03-29T11:19:26.564 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:26.564 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:26.564 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:26.564 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:26.564 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.565 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:26.565 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:26.565 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:26.565 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:26.565 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:26.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2559639541 0 0) 0x7fb4a8070ea0 con 0x7fb4a81457f0 2024-03-29T11:19:26.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb48c004eb0 con 0x7fb4a81457f0 2024-03-29T11:19:26.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb4a0001730 con 0x7fb4a81457f0 2024-03-29T11:19:26.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:26.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 10 monclient(hunting): dump: 2024-03-29T11:19:26.565 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:26.565 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:26.565 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:26.565 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:26.565 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.566 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:26.566 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:26.566 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:26.566 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:26.566 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:26.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1097588318 0 0) 0x7fb48c004eb0 con 0x7fb4a81457f0 2024-03-29T11:19:26.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 >> v1:172.21.15.184:6789/0 conn(0x7fb4a813f650 legacy=0x7fb4a8141340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 >> v1:172.21.15.170:6789/0 conn(0x7fb4a80a9d70 legacy=0x7fb4a8071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:26.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 1 monclient: found mon.c 2024-03-29T11:19:26.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:26.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:26.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a8072020 con 0x7fb4a81457f0 2024-03-29T11:19:26.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:26.567 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:26.567 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb49dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:26.567 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb4af67b640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:19:26.567 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb4af67b640 10 monclient: _renew_subs 2024-03-29T11:19:26.567 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.582+0000 7fb4af67b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:26.567 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.583+0000 7fb4af67b640 1 -- 172.21.15.170:0/372195807 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a81306e0 con 0x7fb4a81457f0 2024-03-29T11:19:26.567 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.583+0000 7fb4af67b640 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:26.567 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.583+0000 7fb4af67b640 10 monclient: _renew_subs 2024-03-29T11:19:26.567 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.583+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4a0001950 con 0x7fb4a81457f0 2024-03-29T11:19:26.567 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.583+0000 7fb4af67b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:26.567 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.583+0000 7fb4af67b640 1 -- 172.21.15.170:0/372195807 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a806f8f0 con 0x7fb4a81457f0 2024-03-29T11:19:26.567 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.583+0000 7fb49dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:26.567 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.583+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb4a0001730 con 0x7fb4a81457f0 2024-03-29T11:19:26.567 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.583+0000 7fb49dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.583+0000 7fb49dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:26.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.583+0000 7fb49dffb640 10 monclient: dump: 2024-03-29T11:19:26.568 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:26.568 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:26.568 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:26.568 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:26.568 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.568 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:26.568 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:26.568 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:26.568 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:26.568 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:26.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.583+0000 7fb49f7fe640 4 set_mon_vals no callback set 2024-03-29T11:19:26.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.584+0000 7fb4af67b640 1 librados: init done 2024-03-29T11:19:26.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.586+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fb4a001aea0 con 0x7fb4a81457f0 2024-03-29T11:19:26.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.587+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fb4a0084bd0 con 0x7fb4a81457f0 2024-03-29T11:19:26.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.587+0000 7fb49dffb640 10 client.4541.objecter ms_dispatch 0x7fb4a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:26.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.587+0000 7fb49dffb640 3 client.4541.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:26.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.587+0000 7fb49dffb640 3 client.4541.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:26.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.587+0000 7fb49dffb640 20 client.4541.objecter dump_active .. 0 homeless 2024-03-29T11:19:26.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.588+0000 7fb4ad417640 10 monclient: get_auth_request con 0x7fb48c06e330 auth_method 0 2024-03-29T11:19:26.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.588+0000 7fb4af67b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.588+0000 7fb4af67b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.588+0000 7fb4af67b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:26.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.588+0000 7fb4af67b640 1 -- 172.21.15.170:0/372195807 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a80713c0 con 0x7fb4a81457f0 2024-03-29T11:19:26.570 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.588+0000 7fb49dffb640 10 client.4541.objecter ms_handle_connect 0x7fb48c06e330 2024-03-29T11:19:26.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.596+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb4a80713c0 con 0x7fb4a81457f0 2024-03-29T11:19:26.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.596+0000 7fb49dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.596+0000 7fb49dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:26.904 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.926+0000 7fb4af67b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:26.904 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.926+0000 7fb4af67b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:26.904 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.926+0000 7fb4af67b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:26.904 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.926+0000 7fb4af67b640 1 -- 172.21.15.170:0/372195807 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4a8075f10 con 0x7fb4a81457f0 2024-03-29T11:19:26.906 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:19:26.906 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:26.906 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.928+0000 7fb49dffb640 1 -- 172.21.15.170:0/372195807 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fb4a80713c0 con 0x7fb4a81457f0 2024-03-29T11:19:26.907 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.928+0000 7fb49dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:26.907 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.928+0000 7fb49dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:26.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.933+0000 7fb4af67b640 10 librados: watch_flush enter 2024-03-29T11:19:26.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.933+0000 7fb4af67b640 10 librados: watch_flush exit 2024-03-29T11:19:26.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.934+0000 7fb4af67b640 20 client.4541.objecter shutdown clearing up homeless session... 2024-03-29T11:19:26.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.934+0000 7fb4af67b640 10 client.4541.objecter successfully canceled tick 2024-03-29T11:19:26.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.934+0000 7fb4af67b640 1 -- 172.21.15.170:0/372195807 >> v1:172.21.15.170:6816/3298994444 conn(0x7fb48c06e330 legacy=0x7fb48c0707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.934+0000 7fb4af67b640 10 monclient: shutdown 2024-03-29T11:19:26.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.934+0000 7fb4af67b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:26.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.934+0000 7fb4af67b640 1 -- 172.21.15.170:0/372195807 >> v1:172.21.15.170:6790/0 conn(0x7fb4a81457f0 legacy=0x7fb4a81e6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.934+0000 7fb4af67b640 1 -- 172.21.15.170:0/372195807 shutdown_connections 2024-03-29T11:19:26.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.934+0000 7fb4af67b640 1 -- 172.21.15.170:0/372195807 >> 172.21.15.170:0/372195807 conn(0x7fb4a8066a00 msgr2=0x7fb4a80b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:26.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.934+0000 7fb4af67b640 1 -- 172.21.15.170:0/372195807 shutdown_connections 2024-03-29T11:19:26.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.934+0000 7fb4af67b640 1 -- 172.21.15.170:0/372195807 wait complete. 2024-03-29T11:19:26.914 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:26.935+0000 7fb4af67b640 1 librados: shutdown 2024-03-29T11:19:26.929 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:33.736 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:33.874 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.896+0000 7f370bd7a640 10 monclient: get_monmap_and_config 2024-03-29T11:19:33.874 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.896+0000 7f370bd7a640 10 monclient: build_initial_monmap 2024-03-29T11:19:33.874 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.896+0000 7f370bd7a640 10 monclient: monmap: 2024-03-29T11:19:33.874 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:19:33.874 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:33.874 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:19:33.898255+0000 2024-03-29T11:19:33.875 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:19:33.898255+0000 2024-03-29T11:19:33.875 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:33.875 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:33.875 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:19:33.875 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:19:33.875 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:19:33.875 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:33.875 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.898+0000 7f370bd7a640 1 Processor -- start 2024-03-29T11:19:33.876 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.898+0000 7f370bd7a640 1 -- start start 2024-03-29T11:19:33.876 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.899+0000 7f370bd7a640 10 monclient: init 2024-03-29T11:19:33.877 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.900+0000 7f370bd7a640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:33.877 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.900+0000 7f370bd7a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:33.878 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.900+0000 7f370bd7a640 10 monclient(hunting): picked mon.noname-c con 0x7f37041457f0 addr v1:172.21.15.184:6789/0 2024-03-29T11:19:33.880 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.901+0000 7f370bd7a640 10 monclient(hunting): picked mon.noname-a con 0x7f370413f650 addr v1:172.21.15.170:6789/0 2024-03-29T11:19:33.880 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.901+0000 7f370bd7a640 10 monclient(hunting): picked mon.noname-b con 0x7f37040a9d70 addr v1:172.21.15.170:6790/0 2024-03-29T11:19:33.880 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.901+0000 7f3709315640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f370413f650 0x7f37040b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:51446/0 (socket says 172.21.15.170:51446) 2024-03-29T11:19:33.880 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.901+0000 7f3709315640 1 -- 172.21.15.170:0/1485139523 learned_addr learned my addr 172.21.15.170:0/1485139523 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:19:33.880 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.901+0000 7f370bd7a640 1 -- 172.21.15.170:0/1485139523 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37040aaef0 con 0x7f370413f650 2024-03-29T11:19:33.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.901+0000 7f370bd7a640 1 -- 172.21.15.170:0/1485139523 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3704145bd0 con 0x7f37041457f0 2024-03-29T11:19:33.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.902+0000 7f370bd7a640 1 -- 172.21.15.170:0/1485139523 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3704145eb0 con 0x7f37040a9d70 2024-03-29T11:19:33.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.902+0000 7f3708b14640 1 -- 172.21.15.170:0/1485139523 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797866165 0 0) 0x7f37040aaef0 con 0x7f370413f650 2024-03-29T11:19:33.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.902+0000 7f370bd7a640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:33.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.902+0000 7f370bd7a640 10 monclient(hunting): authenticate will time out at 1355.214402s 2024-03-29T11:19:33.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.902+0000 7f3708b14640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.902+0000 7f3708b14640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.902+0000 7f3708b14640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.902+0000 7f3708b14640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:19:33.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.902+0000 7f3708b14640 1 -- 172.21.15.170:0/1485139523 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f0003520 con 0x7f370413f650 2024-03-29T11:19:33.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.902+0000 7f3708b14640 1 -- 172.21.15.170:0/1485139523 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2421906586 0 0) 0x7f36f0003520 con 0x7f370413f650 2024-03-29T11:19:33.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.902+0000 7f3708b14640 1 -- 172.21.15.170:0/1485139523 >> v1:172.21.15.170:6790/0 conn(0x7f37040a9d70 legacy=0x7f370414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.902+0000 7f3708b14640 1 -- 172.21.15.170:0/1485139523 >> v1:172.21.15.184:6789/0 conn(0x7f37041457f0 legacy=0x7f370413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:33.882 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.904+0000 7f3708b14640 10 monclient: _finish_hunting 0 2024-03-29T11:19:33.882 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.904+0000 7f3708b14640 1 monclient: found mon.noname-a 2024-03-29T11:19:33.882 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.904+0000 7f3708b14640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:33.883 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.904+0000 7f3708b14640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:19:33.883 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.904+0000 7f3708b14640 1 -- 172.21.15.170:0/1485139523 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3704146340 con 0x7f370413f650 2024-03-29T11:19:33.883 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.904+0000 7f3708b14640 10 monclient: _finish_auth 0 2024-03-29T11:19:33.883 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.904+0000 7f3708b14640 10 monclient: _check_auth_tickets 2024-03-29T11:19:33.883 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.904+0000 7f3708b14640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:33.883 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.904+0000 7f3708b14640 1 -- 172.21.15.170:0/1485139523 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f36fc001a10 con 0x7f370413f650 2024-03-29T11:19:33.883 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.904+0000 7f3708b14640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.883 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.904+0000 7f3708b14640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:33.883 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.904+0000 7f3708b14640 10 monclient: dump: 2024-03-29T11:19:33.883 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:33.883 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:33.883 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:33.883 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:33.883 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.883 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:33.884 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:33.884 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:33.884 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:33.884 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:33.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.904+0000 7f370bd7a640 5 monclient: authenticate success, global_id 4509 2024-03-29T11:19:33.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.904+0000 7f370bd7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:33.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.905+0000 7f3708b14640 1 -- 172.21.15.170:0/1485139523 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36fc001ca0 con 0x7f370413f650 2024-03-29T11:19:33.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.905+0000 7f3708b14640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:33.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.905+0000 7f370bd7a640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:33.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.905+0000 7f370bd7a640 4 set_mon_vals no callback set 2024-03-29T11:19:33.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.905+0000 7f3708b14640 1 -- 172.21.15.170:0/1485139523 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f36fc0020b0 con 0x7f370413f650 2024-03-29T11:19:33.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.905+0000 7f3708b14640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.905+0000 7f3708b14640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:33.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.905+0000 7f3708b14640 10 monclient: dump: 2024-03-29T11:19:33.884 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:33.885 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:33.885 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:33.885 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:33.885 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.885 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:33.885 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:33.885 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:33.885 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:33.885 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:33.893 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.906+0000 7f370bd7a640 10 monclient: shutdown 2024-03-29T11:19:33.893 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.906+0000 7f370bd7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:33.893 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.906+0000 7f370bd7a640 1 -- 172.21.15.170:0/1485139523 >> v1:172.21.15.170:6789/0 conn(0x7f370413f650 legacy=0x7f37040b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.894 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.907+0000 7f370bd7a640 1 -- 172.21.15.170:0/1485139523 shutdown_connections 2024-03-29T11:19:33.894 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.907+0000 7f370bd7a640 1 -- 172.21.15.170:0/1485139523 >> 172.21.15.170:0/1485139523 conn(0x7f3704066a00 msgr2=0x7f3704066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:33.894 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.907+0000 7f370bd7a640 1 -- 172.21.15.170:0/1485139523 shutdown_connections 2024-03-29T11:19:33.894 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.907+0000 7f370bd7a640 1 -- 172.21.15.170:0/1485139523 wait complete. 2024-03-29T11:19:33.894 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.908+0000 7f370bd7a640 10 monclient: build_initial_monmap 2024-03-29T11:19:33.894 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.908+0000 7f370bd7a640 10 monclient: monmap: 2024-03-29T11:19:33.894 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:19:33.894 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:33.894 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:19:33.894 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:19:33.894 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:33.894 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:33.895 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:19:33.895 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:19:33.895 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:19:33.895 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:33.895 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.908+0000 7f370bd7a640 1 librados: starting msgr at 2024-03-29T11:19:33.895 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.908+0000 7f370bd7a640 1 librados: starting objecter 2024-03-29T11:19:33.895 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.908+0000 7f370bd7a640 1 Processor -- start 2024-03-29T11:19:33.895 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.908+0000 7f370bd7a640 1 -- start start 2024-03-29T11:19:33.895 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.908+0000 7f370bd7a640 1 librados: setting wanted keys 2024-03-29T11:19:33.895 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.908+0000 7f370bd7a640 1 librados: calling monclient init 2024-03-29T11:19:33.895 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.908+0000 7f370bd7a640 10 monclient: init 2024-03-29T11:19:33.895 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.909+0000 7f370bd7a640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:33.895 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.909+0000 7f370bd7a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:33.895 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.909+0000 7f370bd7a640 10 monclient(hunting): picked mon.noname-a con 0x7f37040a9d70 addr v1:172.21.15.170:6789/0 2024-03-29T11:19:33.895 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f370bd7a640 10 monclient(hunting): picked mon.noname-c con 0x7f370413f650 addr v1:172.21.15.170:6790/0 2024-03-29T11:19:33.896 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f370bd7a640 10 monclient(hunting): picked mon.noname-b con 0x7f37041457f0 addr v1:172.21.15.184:6789/0 2024-03-29T11:19:33.896 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f370bd7a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37041ccae0 con 0x7f37040a9d70 2024-03-29T11:19:33.896 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f370bd7a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37041ccd50 con 0x7f37041457f0 2024-03-29T11:19:33.896 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f370bd7a640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37041cd030 con 0x7f370413f650 2024-03-29T11:19:33.896 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f370bd7a640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:33.896 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f370bd7a640 10 monclient(hunting): authenticate will time out at 1355.222089s 2024-03-29T11:19:33.896 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f3709b16640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f37040a9d70 0x7f3704137dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:43132/0 (socket says 172.21.15.170:43132) 2024-03-29T11:19:33.896 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f3709b16640 1 -- 172.21.15.170:0/1104186691 learned_addr learned my addr 172.21.15.170:0/1104186691 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:19:33.896 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f36fa7fc640 10 client.?.objecter ms_handle_connect 0x7f37040a9d70 2024-03-29T11:19:33.896 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f36fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.896 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f36fa7fc640 10 client.?.objecter ms_handle_connect 0x7f370413f650 2024-03-29T11:19:33.896 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f36fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.896 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807469977 0 0) 0x7f37041cd030 con 0x7f370413f650 2024-03-29T11:19:33.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f36fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f36fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f36fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f36fa7fc640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:19:33.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e8004eb0 con 0x7f370413f650 2024-03-29T11:19:33.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263129654 0 0) 0x7f37041ccae0 con 0x7f37040a9d70 2024-03-29T11:19:33.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f36fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f36fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f36fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f36fa7fc640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:19:33.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.910+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37041cd030 con 0x7f37040a9d70 2024-03-29T11:19:33.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.911+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 528398314 0 0) 0x7f36e8004eb0 con 0x7f370413f650 2024-03-29T11:19:33.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.911+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37041ccae0 con 0x7f370413f650 2024-03-29T11:19:33.898 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.911+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f36fc001830 con 0x7f370413f650 2024-03-29T11:19:33.898 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.911+0000 7f36fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.898 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.911+0000 7f36fa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:33.898 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.911+0000 7f36fa7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:33.898 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:33.898 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:33.898 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:33.898 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:33.898 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.898 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:33.898 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:33.898 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:33.898 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:33.899 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:33.899 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.911+0000 7f36fa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:33.899 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.911+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1940877941 0 0) 0x7f37041cd030 con 0x7f37040a9d70 2024-03-29T11:19:33.899 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.911+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36e8004eb0 con 0x7f37040a9d70 2024-03-29T11:19:33.899 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.911+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f36f4001790 con 0x7f37040a9d70 2024-03-29T11:19:33.899 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.911+0000 7f36fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.899 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.911+0000 7f36fa7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:33.899 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.911+0000 7f36fa7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:33.899 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:33.899 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:33.899 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:33.899 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:33.899 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.899 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:33.899 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:33.899 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:33.899 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:33.900 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:33.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.911+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2512944110 0 0) 0x7f36e8004eb0 con 0x7f37040a9d70 2024-03-29T11:19:33.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.911+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 >> v1:172.21.15.170:6790/0 conn(0x7f370413f650 legacy=0x7f37041cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.911+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 >> v1:172.21.15.184:6789/0 conn(0x7f37041457f0 legacy=0x7f37041e2c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:33.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.912+0000 7f36fa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:33.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.912+0000 7f36fa7fc640 1 monclient: found mon.a 2024-03-29T11:19:33.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.912+0000 7f36fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:33.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.912+0000 7f36fa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:19:33.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.912+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37041cb340 con 0x7f37040a9d70 2024-03-29T11:19:33.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.912+0000 7f36fa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:33.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.912+0000 7f36fa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:33.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.912+0000 7f36fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:33.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.912+0000 7f370bd7a640 5 monclient: authenticate success, global_id 4512 2024-03-29T11:19:33.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.912+0000 7f370bd7a640 10 monclient: _renew_subs 2024-03-29T11:19:33.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.912+0000 7f370bd7a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:19:33.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.912+0000 7f370bd7a640 1 -- 172.21.15.170:0/1104186691 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37041cb620 con 0x7f37040a9d70 2024-03-29T11:19:33.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.912+0000 7f370bd7a640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:33.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.912+0000 7f370bd7a640 10 monclient: _renew_subs 2024-03-29T11:19:33.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.912+0000 7f370bd7a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:19:33.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.912+0000 7f370bd7a640 1 -- 172.21.15.170:0/1104186691 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37041cbe10 con 0x7f37040a9d70 2024-03-29T11:19:33.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.912+0000 7f370bd7a640 1 librados: init done 2024-03-29T11:19:33.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.913+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36f4001ca0 con 0x7f37040a9d70 2024-03-29T11:19:33.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.913+0000 7f36fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:33.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.913+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f36f4004a00 con 0x7f37040a9d70 2024-03-29T11:19:33.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.913+0000 7f36fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.913+0000 7f36fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:33.902 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.913+0000 7f36fa7fc640 10 monclient: dump: 2024-03-29T11:19:33.902 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:33.902 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:33.902 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:33.902 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:33.902 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.902 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:33.902 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:33.902 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:33.902 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:33.902 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:33.902 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.913+0000 7f36fbfff640 4 set_mon_vals no callback set 2024-03-29T11:19:33.902 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.914+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f36f401a9d0 con 0x7f37040a9d70 2024-03-29T11:19:33.902 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.915+0000 7f370bd7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.915+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f36f40849c0 con 0x7f37040a9d70 2024-03-29T11:19:33.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.915+0000 7f36fa7fc640 10 client.4512.objecter ms_dispatch 0x7f3704000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:33.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.915+0000 7f36fa7fc640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:33.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.915+0000 7f36fa7fc640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:33.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.915+0000 7f370bd7a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.915+0000 7f370bd7a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:19:33.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.915+0000 7f36fa7fc640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-29T11:19:33.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.915+0000 7f370bd7a640 1 -- 172.21.15.170:0/1104186691 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36ec005080 con 0x7f37040a9d70 2024-03-29T11:19:33.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.915+0000 7f3709315640 10 monclient: get_auth_request con 0x7f36e806df10 auth_method 0 2024-03-29T11:19:33.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.915+0000 7f36fa7fc640 10 client.4512.objecter ms_handle_connect 0x7f36e806df10 2024-03-29T11:19:33.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.923+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36f40500d0 con 0x7f37040a9d70 2024-03-29T11:19:33.904 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.923+0000 7f36fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.904 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:33.923+0000 7f36fa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:34.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.245+0000 7f370bd7a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:34.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.245+0000 7f370bd7a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:34.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.245+0000 7f370bd7a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:19:34.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.245+0000 7f370bd7a640 1 -- 172.21.15.170:0/1104186691 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36ec005330 con 0x7f37040a9d70 2024-03-29T11:19:34.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.247+0000 7f36fa7fc640 1 -- 172.21.15.170:0/1104186691 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f36f40521d0 con 0x7f37040a9d70 2024-03-29T11:19:34.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.247+0000 7f36fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:34.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.247+0000 7f36fa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:34.226 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:19:34.227 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:34.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.252+0000 7f370bd7a640 10 librados: watch_flush enter 2024-03-29T11:19:34.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.252+0000 7f370bd7a640 10 librados: watch_flush exit 2024-03-29T11:19:34.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.252+0000 7f370bd7a640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-29T11:19:34.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.252+0000 7f370bd7a640 10 client.4512.objecter successfully canceled tick 2024-03-29T11:19:34.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.252+0000 7f370bd7a640 1 -- 172.21.15.170:0/1104186691 >> v1:172.21.15.170:6816/3298994444 conn(0x7f36e806df10 legacy=0x7f36e80703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.252+0000 7f370bd7a640 10 monclient: shutdown 2024-03-29T11:19:34.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.252+0000 7f370bd7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:34.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.252+0000 7f370bd7a640 1 -- 172.21.15.170:0/1104186691 >> v1:172.21.15.170:6789/0 conn(0x7f37040a9d70 legacy=0x7f3704137dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.252+0000 7f370bd7a640 1 -- 172.21.15.170:0/1104186691 shutdown_connections 2024-03-29T11:19:34.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.253+0000 7f370bd7a640 1 -- 172.21.15.170:0/1104186691 >> 172.21.15.170:0/1104186691 conn(0x7f3704066a00 msgr2=0x7f3704066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:34.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.253+0000 7f370bd7a640 1 -- 172.21.15.170:0/1104186691 shutdown_connections 2024-03-29T11:19:34.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.253+0000 7f370bd7a640 1 -- 172.21.15.170:0/1104186691 wait complete. 2024-03-29T11:19:34.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:34.253+0000 7f370bd7a640 1 librados: shutdown 2024-03-29T11:19:34.248 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:41.055 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:41.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.230+0000 7ff1a140f640 10 monclient: get_monmap_and_config 2024-03-29T11:19:41.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.230+0000 7ff1a140f640 10 monclient: build_initial_monmap 2024-03-29T11:19:41.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.230+0000 7ff1a140f640 10 monclient: monmap: 2024-03-29T11:19:41.208 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:19:41.208 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:41.208 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:19:41.231845+0000 2024-03-29T11:19:41.208 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:19:41.231845+0000 2024-03-29T11:19:41.208 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:41.208 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:41.208 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:19:41.208 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:19:41.208 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:19:41.208 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:41.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.231+0000 7ff1a140f640 1 Processor -- start 2024-03-29T11:19:41.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.231+0000 7ff1a140f640 1 -- start start 2024-03-29T11:19:41.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.231+0000 7ff1a140f640 10 monclient: init 2024-03-29T11:19:41.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.232+0000 7ff1a140f640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:41.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.232+0000 7ff1a140f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:41.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.232+0000 7ff1a140f640 10 monclient(hunting): picked mon.noname-b con 0x7ff19c1457f0 addr v1:172.21.15.170:6790/0 2024-03-29T11:19:41.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.232+0000 7ff1a140f640 10 monclient(hunting): picked mon.noname-c con 0x7ff19c13f650 addr v1:172.21.15.184:6789/0 2024-03-29T11:19:41.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.232+0000 7ff1a140f640 10 monclient(hunting): picked mon.noname-a con 0x7ff19c0a9d70 addr v1:172.21.15.170:6789/0 2024-03-29T11:19:41.210 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.232+0000 7ff1a140f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c0aaef0 con 0x7ff19c0a9d70 2024-03-29T11:19:41.210 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.232+0000 7ff1a140f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c145bd0 con 0x7ff19c13f650 2024-03-29T11:19:41.210 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.232+0000 7ff1a140f640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c145eb0 con 0x7ff19c1457f0 2024-03-29T11:19:41.210 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.232+0000 7ff1a140f640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:41.210 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.232+0000 7ff1a140f640 10 monclient(hunting): authenticate will time out at 1362.544324s 2024-03-29T11:19:41.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.232+0000 7ff19a7fc640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7ff19c13f650 0x7ff19c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:54770/0 (socket says 172.21.15.170:54770) 2024-03-29T11:19:41.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.232+0000 7ff19a7fc640 1 -- 172.21.15.170:0/139191064 learned_addr learned my addr 172.21.15.170:0/139191064 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:19:41.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.233+0000 7ff199ffb640 1 -- 172.21.15.170:0/139191064 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199607120 0 0) 0x7ff19c145eb0 con 0x7ff19c1457f0 2024-03-29T11:19:41.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.233+0000 7ff199ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.233+0000 7ff199ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.233+0000 7ff199ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.233+0000 7ff199ffb640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:19:41.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.233+0000 7ff199ffb640 1 -- 172.21.15.170:0/139191064 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff188003520 con 0x7ff19c1457f0 2024-03-29T11:19:41.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.233+0000 7ff199ffb640 1 -- 172.21.15.170:0/139191064 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893517609 0 0) 0x7ff19c145bd0 con 0x7ff19c13f650 2024-03-29T11:19:41.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.233+0000 7ff199ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.233+0000 7ff199ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.233+0000 7ff199ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.233+0000 7ff199ffb640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:19:41.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.233+0000 7ff199ffb640 1 -- 172.21.15.170:0/139191064 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff19c145eb0 con 0x7ff19c13f650 2024-03-29T11:19:41.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.233+0000 7ff199ffb640 1 -- 172.21.15.170:0/139191064 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045173164 0 0) 0x7ff19c0aaef0 con 0x7ff19c0a9d70 2024-03-29T11:19:41.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.233+0000 7ff199ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.233+0000 7ff199ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.233+0000 7ff199ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.233+0000 7ff199ffb640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:19:41.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.233+0000 7ff199ffb640 1 -- 172.21.15.170:0/139191064 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff19c145bd0 con 0x7ff19c0a9d70 2024-03-29T11:19:41.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.234+0000 7ff199ffb640 1 -- 172.21.15.170:0/139191064 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1952418434 0 0) 0x7ff188003520 con 0x7ff19c1457f0 2024-03-29T11:19:41.216 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.234+0000 7ff199ffb640 1 -- 172.21.15.170:0/139191064 >> v1:172.21.15.184:6789/0 conn(0x7ff19c13f650 legacy=0x7ff19c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.216 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.234+0000 7ff199ffb640 1 -- 172.21.15.170:0/139191064 >> v1:172.21.15.170:6789/0 conn(0x7ff19c0a9d70 legacy=0x7ff19c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.216 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.234+0000 7ff199ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:41.216 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.234+0000 7ff199ffb640 1 monclient: found mon.noname-b 2024-03-29T11:19:41.216 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.234+0000 7ff199ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:41.216 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.234+0000 7ff199ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:19:41.216 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.234+0000 7ff199ffb640 1 -- 172.21.15.170:0/139191064 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c146340 con 0x7ff19c1457f0 2024-03-29T11:19:41.216 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.234+0000 7ff199ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:41.216 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.234+0000 7ff199ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:41.216 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.234+0000 7ff199ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:41.216 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.234+0000 7ff199ffb640 1 -- 172.21.15.170:0/139191064 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7ff184001690 con 0x7ff19c1457f0 2024-03-29T11:19:41.216 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.234+0000 7ff1a140f640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:19:41.216 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.234+0000 7ff1a140f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:41.216 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.234+0000 7ff199ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.217 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.234+0000 7ff199ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:41.217 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.234+0000 7ff199ffb640 10 monclient: dump: 2024-03-29T11:19:41.217 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:41.217 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:41.217 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:41.217 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:41.217 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.217 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:41.217 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:41.217 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:41.217 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:41.217 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:41.217 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.234+0000 7ff1a140f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:41.217 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.235+0000 7ff199ffb640 1 -- 172.21.15.170:0/139191064 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1840018b0 con 0x7ff19c1457f0 2024-03-29T11:19:41.217 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.235+0000 7ff199ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:41.218 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.235+0000 7ff1a140f640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:41.218 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.235+0000 7ff1a140f640 4 set_mon_vals no callback set 2024-03-29T11:19:41.218 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.236+0000 7ff1a140f640 10 monclient: shutdown 2024-03-29T11:19:41.218 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.236+0000 7ff1a140f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:41.218 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.236+0000 7ff1a140f640 1 -- 172.21.15.170:0/139191064 >> v1:172.21.15.170:6790/0 conn(0x7ff19c1457f0 legacy=0x7ff19c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.218 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.236+0000 7ff199ffb640 1 -- 172.21.15.170:0/139191064 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7ff184001c40 con 0x7ff19c1457f0 2024-03-29T11:19:41.218 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.236+0000 7ff1a140f640 1 -- 172.21.15.170:0/139191064 shutdown_connections 2024-03-29T11:19:41.218 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.236+0000 7ff1a140f640 1 -- 172.21.15.170:0/139191064 >> 172.21.15.170:0/139191064 conn(0x7ff19c066a00 msgr2=0x7ff19c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:41.218 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.236+0000 7ff199ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:19:41.218 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.236+0000 7ff1a140f640 1 -- 172.21.15.170:0/139191064 shutdown_connections 2024-03-29T11:19:41.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.237+0000 7ff1a140f640 1 -- 172.21.15.170:0/139191064 wait complete. 2024-03-29T11:19:41.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.238+0000 7ff1a140f640 10 monclient: build_initial_monmap 2024-03-29T11:19:41.221 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.238+0000 7ff1a140f640 10 monclient: monmap: 2024-03-29T11:19:41.221 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:19:41.221 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:41.221 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:19:41.221 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:19:41.221 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:41.221 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:41.221 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:19:41.221 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:19:41.221 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:19:41.221 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:41.221 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.238+0000 7ff1a140f640 1 librados: starting msgr at 2024-03-29T11:19:41.221 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.238+0000 7ff1a140f640 1 librados: starting objecter 2024-03-29T11:19:41.221 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.238+0000 7ff1a140f640 1 Processor -- start 2024-03-29T11:19:41.221 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.239+0000 7ff1a140f640 1 -- start start 2024-03-29T11:19:41.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.239+0000 7ff1a140f640 1 librados: setting wanted keys 2024-03-29T11:19:41.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.239+0000 7ff1a140f640 1 librados: calling monclient init 2024-03-29T11:19:41.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.239+0000 7ff1a140f640 10 monclient: init 2024-03-29T11:19:41.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.239+0000 7ff1a140f640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:41.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.239+0000 7ff1a140f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:41.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.239+0000 7ff1a140f640 10 monclient(hunting): picked mon.noname-a con 0x7ff19c0a9d70 addr v1:172.21.15.170:6789/0 2024-03-29T11:19:41.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.239+0000 7ff1a140f640 10 monclient(hunting): picked mon.noname-b con 0x7ff19c13f650 addr v1:172.21.15.184:6789/0 2024-03-29T11:19:41.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.240+0000 7ff1a140f640 10 monclient(hunting): picked mon.noname-c con 0x7ff19c0707c0 addr v1:172.21.15.170:6790/0 2024-03-29T11:19:41.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.240+0000 7ff1a140f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c1d8d20 con 0x7ff19c0a9d70 2024-03-29T11:19:41.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.240+0000 7ff1a140f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c070330 con 0x7ff19c13f650 2024-03-29T11:19:41.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.240+0000 7ff1a140f640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c071c60 con 0x7ff19c0707c0 2024-03-29T11:19:41.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.240+0000 7ff1a140f640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:41.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.240+0000 7ff1a140f640 10 monclient(hunting): authenticate will time out at 1362.552063s 2024-03-29T11:19:41.222 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.240+0000 7ff19a7fc640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7ff19c13f650 0x7ff19c0b0880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:55308/0 (socket says 172.21.15.170:55308) 2024-03-29T11:19:41.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.240+0000 7ff19a7fc640 1 -- 172.21.15.170:0/535107506 learned_addr learned my addr 172.21.15.170:0/535107506 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:19:41.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.240+0000 7ff1937fe640 10 client.?.objecter ms_handle_connect 0x7ff19c13f650 2024-03-29T11:19:41.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.240+0000 7ff1937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:41.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.240+0000 7ff1937fe640 10 client.?.objecter ms_handle_connect 0x7ff19c0707c0 2024-03-29T11:19:41.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.240+0000 7ff1937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:41.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 10 client.?.objecter ms_handle_connect 0x7ff19c0a9d70 2024-03-29T11:19:41.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:41.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139162743 0 0) 0x7ff19c071c60 con 0x7ff19c0707c0 2024-03-29T11:19:41.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:19:41.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff17c004eb0 con 0x7ff19c0707c0 2024-03-29T11:19:41.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388366085 0 0) 0x7ff19c146340 con 0x7ff19c0a9d70 2024-03-29T11:19:41.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.223 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:19:41.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff19c071c60 con 0x7ff19c0a9d70 2024-03-29T11:19:41.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278743726 0 0) 0x7ff19c070330 con 0x7ff19c13f650 2024-03-29T11:19:41.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:19:41.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff19c146340 con 0x7ff19c13f650 2024-03-29T11:19:41.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 408100178 0 0) 0x7ff19c146340 con 0x7ff19c13f650 2024-03-29T11:19:41.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.241+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff19c070330 con 0x7ff19c13f650 2024-03-29T11:19:41.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7ff18c001bd0 con 0x7ff19c13f650 2024-03-29T11:19:41.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:41.224 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 10 monclient(hunting): dump: 2024-03-29T11:19:41.224 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:41.224 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:41.224 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:41.225 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:41.225 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.225 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:41.225 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:41.225 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:41.225 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:41.225 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:41.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:41.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3758572232 0 0) 0x7ff19c071c60 con 0x7ff19c0a9d70 2024-03-29T11:19:41.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff19c146340 con 0x7ff19c0a9d70 2024-03-29T11:19:41.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7ff184001a20 con 0x7ff19c0a9d70 2024-03-29T11:19:41.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:41.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 10 monclient(hunting): dump: 2024-03-29T11:19:41.225 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:41.225 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:41.226 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:41.226 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:41.226 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.226 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:41.226 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:41.226 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:41.226 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:41.226 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:41.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4008086079 0 0) 0x7ff19c070330 con 0x7ff19c13f650 2024-03-29T11:19:41.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 >> v1:172.21.15.170:6790/0 conn(0x7ff19c0707c0 legacy=0x7ff19c0b0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 >> v1:172.21.15.170:6789/0 conn(0x7ff19c0a9d70 legacy=0x7ff19c1de600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:41.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 1 monclient: found mon.b 2024-03-29T11:19:41.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:41.226 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:19:41.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c06ffc0 con 0x7ff19c13f650 2024-03-29T11:19:41.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:41.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:41.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:41.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1a140f640 5 monclient: authenticate success, global_id 4486 2024-03-29T11:19:41.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1a140f640 10 monclient: _renew_subs 2024-03-29T11:19:41.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1a140f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:19:41.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1a140f640 1 -- 172.21.15.170:0/535107506 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff19c1ded10 con 0x7ff19c13f650 2024-03-29T11:19:41.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1a140f640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:41.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.242+0000 7ff1a140f640 10 monclient: _renew_subs 2024-03-29T11:19:41.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.243+0000 7ff1a140f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:19:41.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.243+0000 7ff1a140f640 1 -- 172.21.15.170:0/535107506 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff19c1421f0 con 0x7ff19c13f650 2024-03-29T11:19:41.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.243+0000 7ff1a140f640 1 librados: init done 2024-03-29T11:19:41.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.243+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff18c004b60 con 0x7ff19c13f650 2024-03-29T11:19:41.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.243+0000 7ff1937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:41.227 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.243+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7ff18c004ef0 con 0x7ff19c13f650 2024-03-29T11:19:41.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.243+0000 7ff1997fa640 4 set_mon_vals no callback set 2024-03-29T11:19:41.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.243+0000 7ff1937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.243+0000 7ff1937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:41.228 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.243+0000 7ff1937fe640 10 monclient: dump: 2024-03-29T11:19:41.228 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:41.228 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:41.228 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:41.228 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:41.228 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.228 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:41.228 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:41.228 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:41.228 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:41.228 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:41.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.244+0000 7ff1a140f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:41.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.244+0000 7ff1a140f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:41.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.244+0000 7ff1a140f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:19:41.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.244+0000 7ff1a140f640 1 -- 172.21.15.170:0/535107506 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff180005080 con 0x7ff19c13f650 2024-03-29T11:19:41.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.246+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7ff18c005050 con 0x7ff19c13f650 2024-03-29T11:19:41.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.247+0000 7ff19affd640 10 monclient: get_auth_request con 0x7ff17c06ddb0 auth_method 0 2024-03-29T11:19:41.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.247+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7ff18c095c60 con 0x7ff19c13f650 2024-03-29T11:19:41.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.247+0000 7ff1937fe640 10 client.4486.objecter ms_dispatch 0x7ff19c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:41.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.247+0000 7ff1937fe640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:41.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.247+0000 7ff1937fe640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:41.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.247+0000 7ff1937fe640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-29T11:19:41.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.247+0000 7ff1937fe640 10 client.4486.objecter ms_handle_connect 0x7ff17c06ddb0 2024-03-29T11:19:41.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.253+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff18c061540 con 0x7ff19c13f650 2024-03-29T11:19:41.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.253+0000 7ff1937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:41.231 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.253+0000 7ff1937fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:41.618 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.640+0000 7ff1a140f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:41.619 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.640+0000 7ff1a140f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:41.619 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.640+0000 7ff1a140f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:19:41.619 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.640+0000 7ff1a140f640 1 -- 172.21.15.170:0/535107506 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff180005330 con 0x7ff19c13f650 2024-03-29T11:19:41.619 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.641+0000 7ff1937fe640 1 -- 172.21.15.170:0/535107506 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7ff18c063470 con 0x7ff19c13f650 2024-03-29T11:19:41.619 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.641+0000 7ff1937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:41.619 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.641+0000 7ff1937fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:41.619 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:19:41.619 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:41.622 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.645+0000 7ff1a140f640 10 librados: watch_flush enter 2024-03-29T11:19:41.622 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.645+0000 7ff1a140f640 10 librados: watch_flush exit 2024-03-29T11:19:41.622 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.645+0000 7ff1a140f640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-29T11:19:41.623 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.645+0000 7ff1a140f640 10 client.4486.objecter successfully canceled tick 2024-03-29T11:19:41.623 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.645+0000 7ff1a140f640 1 -- 172.21.15.170:0/535107506 >> v1:172.21.15.170:6816/3298994444 conn(0x7ff17c06ddb0 legacy=0x7ff17c070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.623 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.645+0000 7ff1a140f640 10 monclient: shutdown 2024-03-29T11:19:41.623 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.645+0000 7ff1a140f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:41.623 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.645+0000 7ff1a140f640 1 -- 172.21.15.170:0/535107506 >> v1:172.21.15.184:6789/0 conn(0x7ff19c13f650 legacy=0x7ff19c0b0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.623 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.645+0000 7ff1a140f640 1 -- 172.21.15.170:0/535107506 shutdown_connections 2024-03-29T11:19:41.623 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.645+0000 7ff1a140f640 1 -- 172.21.15.170:0/535107506 >> 172.21.15.170:0/535107506 conn(0x7ff19c066a00 msgr2=0x7ff19c13c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:41.624 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.646+0000 7ff1a140f640 1 -- 172.21.15.170:0/535107506 shutdown_connections 2024-03-29T11:19:41.624 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.646+0000 7ff1a140f640 1 -- 172.21.15.170:0/535107506 wait complete. 2024-03-29T11:19:41.624 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:41.646+0000 7ff1a140f640 1 librados: shutdown 2024-03-29T11:19:41.638 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:48.445 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:48.578 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.602+0000 7fb4ae546640 10 monclient: get_monmap_and_config 2024-03-29T11:19:48.579 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.602+0000 7fb4ae546640 10 monclient: build_initial_monmap 2024-03-29T11:19:48.579 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.602+0000 7fb4ae546640 10 monclient: monmap: 2024-03-29T11:19:48.579 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:19:48.579 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:48.579 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:19:48.603163+0000 2024-03-29T11:19:48.579 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:19:48.603163+0000 2024-03-29T11:19:48.579 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:48.579 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:48.579 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:19:48.579 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:19:48.579 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:19:48.579 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:48.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.603+0000 7fb4ae546640 1 Processor -- start 2024-03-29T11:19:48.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.603+0000 7fb4ae546640 1 -- start start 2024-03-29T11:19:48.580 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.603+0000 7fb4ae546640 10 monclient: init 2024-03-29T11:19:48.581 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.604+0000 7fb4ae546640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:48.581 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.604+0000 7fb4ae546640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:48.581 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.604+0000 7fb4ae546640 10 monclient(hunting): picked mon.noname-a con 0x7fb4a80b4710 addr v1:172.21.15.170:6789/0 2024-03-29T11:19:48.581 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.604+0000 7fb4ae546640 10 monclient(hunting): picked mon.noname-c con 0x7fb4a80b3970 addr v1:172.21.15.184:6789/0 2024-03-29T11:19:48.581 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.604+0000 7fb4ae546640 10 monclient(hunting): picked mon.noname-b con 0x7fb4a813f390 addr v1:172.21.15.170:6790/0 2024-03-29T11:19:48.581 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.604+0000 7fb4ae546640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a80af020 con 0x7fb4a80b4710 2024-03-29T11:19:48.582 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.604+0000 7fb4ae546640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a80b0910 con 0x7fb4a80b3970 2024-03-29T11:19:48.582 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.604+0000 7fb4ae546640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a80b0bf0 con 0x7fb4a813f390 2024-03-29T11:19:48.582 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.604+0000 7fb4ae546640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:48.582 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.604+0000 7fb4ae546640 10 monclient(hunting): authenticate will time out at 1369.915834s 2024-03-29T11:19:48.582 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.604+0000 7fb4acae3640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fb4a813f390 0x7fb4a813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:51952/0 (socket says 172.21.15.170:51952) 2024-03-29T11:19:48.582 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.604+0000 7fb4acae3640 1 -- 172.21.15.170:0/2800099292 learned_addr learned my addr 172.21.15.170:0/2800099292 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:19:48.582 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.605+0000 7fb4a6ffd640 1 -- 172.21.15.170:0/2800099292 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266956110 0 0) 0x7fb4a80b0bf0 con 0x7fb4a813f390 2024-03-29T11:19:48.582 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.605+0000 7fb4a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:48.582 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.605+0000 7fb4a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:48.582 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.605+0000 7fb4a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:48.582 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.605+0000 7fb4a6ffd640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:19:48.582 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.605+0000 7fb4a6ffd640 1 -- 172.21.15.170:0/2800099292 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb498003520 con 0x7fb4a813f390 2024-03-29T11:19:48.582 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.605+0000 7fb4a6ffd640 1 -- 172.21.15.170:0/2800099292 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 863056958 0 0) 0x7fb498003520 con 0x7fb4a813f390 2024-03-29T11:19:48.582 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.605+0000 7fb4a6ffd640 1 -- 172.21.15.170:0/2800099292 >> v1:172.21.15.184:6789/0 conn(0x7fb4a80b3970 legacy=0x7fb4a80af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:48.583 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.605+0000 7fb4a6ffd640 1 -- 172.21.15.170:0/2800099292 >> v1:172.21.15.170:6789/0 conn(0x7fb4a80b4710 legacy=0x7fb4a80b1010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:48.584 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.607+0000 7fb4a6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:48.584 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.607+0000 7fb4a6ffd640 1 monclient: found mon.noname-b 2024-03-29T11:19:48.584 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:48.584 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4a6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:19:48.584 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4a6ffd640 1 -- 172.21.15.170:0/2800099292 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a80aeb00 con 0x7fb4a813f390 2024-03-29T11:19:48.584 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4a6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:48.585 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4a6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:48.585 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:48.585 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4ae546640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:19:48.585 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4ae546640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:48.585 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4a6ffd640 1 -- 172.21.15.170:0/2800099292 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb4a0001a10 con 0x7fb4a813f390 2024-03-29T11:19:48.585 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:48.585 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4a6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:48.585 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4a6ffd640 10 monclient: dump: 2024-03-29T11:19:48.585 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:48.585 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:48.585 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:48.585 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:48.585 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:48.585 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:48.586 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:48.586 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:48.586 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:48.586 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:48.586 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4a6ffd640 1 -- 172.21.15.170:0/2800099292 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4a0001ca0 con 0x7fb4a813f390 2024-03-29T11:19:48.586 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4ae546640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:48.587 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:48.587 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4a6ffd640 1 -- 172.21.15.170:0/2800099292 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb4a0002030 con 0x7fb4a813f390 2024-03-29T11:19:48.587 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:48.587 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:48.587 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4a6ffd640 10 monclient: dump: 2024-03-29T11:19:48.587 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:48.587 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:48.587 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:48.588 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:48.588 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:48.588 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:48.588 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:48.588 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:48.588 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:48.588 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:48.588 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4ae546640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:48.588 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.608+0000 7fb4ae546640 4 set_mon_vals no callback set 2024-03-29T11:19:48.588 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.609+0000 7fb4ae546640 10 monclient: shutdown 2024-03-29T11:19:48.588 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.609+0000 7fb4ae546640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:48.588 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.609+0000 7fb4ae546640 1 -- 172.21.15.170:0/2800099292 >> v1:172.21.15.170:6790/0 conn(0x7fb4a813f390 legacy=0x7fb4a813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:48.588 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.609+0000 7fb4ae546640 1 -- 172.21.15.170:0/2800099292 shutdown_connections 2024-03-29T11:19:48.588 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.609+0000 7fb4ae546640 1 -- 172.21.15.170:0/2800099292 >> 172.21.15.170:0/2800099292 conn(0x7fb4a8066a00 msgr2=0x7fb4a8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:48.589 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.609+0000 7fb4ae546640 1 -- 172.21.15.170:0/2800099292 shutdown_connections 2024-03-29T11:19:48.589 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.609+0000 7fb4ae546640 1 -- 172.21.15.170:0/2800099292 wait complete. 2024-03-29T11:19:48.589 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.610+0000 7fb4ae546640 10 monclient: build_initial_monmap 2024-03-29T11:19:48.589 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.610+0000 7fb4ae546640 10 monclient: monmap: 2024-03-29T11:19:48.589 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:19:48.589 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:48.589 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:19:48.589 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:19:48.589 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:48.589 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:48.589 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:19:48.589 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:19:48.589 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:19:48.589 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:48.590 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.611+0000 7fb4ae546640 1 librados: starting msgr at 2024-03-29T11:19:48.590 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.611+0000 7fb4ae546640 1 librados: starting objecter 2024-03-29T11:19:48.590 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.611+0000 7fb4ae546640 1 Processor -- start 2024-03-29T11:19:48.592 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.611+0000 7fb4ae546640 1 -- start start 2024-03-29T11:19:48.592 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.611+0000 7fb4ae546640 1 librados: setting wanted keys 2024-03-29T11:19:48.592 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.611+0000 7fb4ae546640 1 librados: calling monclient init 2024-03-29T11:19:48.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.611+0000 7fb4ae546640 10 monclient: init 2024-03-29T11:19:48.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.612+0000 7fb4ae546640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:48.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.612+0000 7fb4ae546640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:48.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.612+0000 7fb4ae546640 10 monclient(hunting): picked mon.noname-a con 0x7fb4a80b3970 addr v1:172.21.15.170:6789/0 2024-03-29T11:19:48.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.612+0000 7fb4ae546640 10 monclient(hunting): picked mon.noname-c con 0x7fb4a80b4710 addr v1:172.21.15.170:6790/0 2024-03-29T11:19:48.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.612+0000 7fb4ae546640 10 monclient(hunting): picked mon.noname-b con 0x7fb4a813f390 addr v1:172.21.15.184:6789/0 2024-03-29T11:19:48.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.612+0000 7fb4ae546640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a814daa0 con 0x7fb4a80b3970 2024-03-29T11:19:48.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.612+0000 7fb4ae546640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a814e500 con 0x7fb4a813f390 2024-03-29T11:19:48.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.612+0000 7fb4ae546640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a814d730 con 0x7fb4a80b4710 2024-03-29T11:19:48.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.612+0000 7fb4ae546640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:48.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.612+0000 7fb4ae546640 10 monclient(hunting): authenticate will time out at 1369.923999s 2024-03-29T11:19:48.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.613+0000 7fb4a77fe640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fb4a80b4710 0x7fb4a81425b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:53284/0 (socket says 172.21.15.170:53284) 2024-03-29T11:19:48.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.613+0000 7fb4a77fe640 1 -- 172.21.15.170:0/3002366570 learned_addr learned my addr 172.21.15.170:0/3002366570 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:19:48.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.613+0000 7fb4a4ff9640 10 client.?.objecter ms_handle_connect 0x7fb4a80b4710 2024-03-29T11:19:48.593 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.613+0000 7fb4a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:48.594 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.613+0000 7fb4a4ff9640 1 -- 172.21.15.170:0/3002366570 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779495598 0 0) 0x7fb4a814d730 con 0x7fb4a80b4710 2024-03-29T11:19:48.594 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.613+0000 7fb4a4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:48.594 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.613+0000 7fb4a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:48.594 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.613+0000 7fb4a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:48.594 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.613+0000 7fb4a4ff9640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:19:48.594 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.613+0000 7fb4a4ff9640 1 -- 172.21.15.170:0/3002366570 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48c004ee0 con 0x7fb4a80b4710 2024-03-29T11:19:48.594 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.614+0000 7fb4a4ff9640 1 -- 172.21.15.170:0/3002366570 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3213378397 0 0) 0x7fb48c004ee0 con 0x7fb4a80b4710 2024-03-29T11:19:48.594 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.614+0000 7fb4a4ff9640 1 -- 172.21.15.170:0/3002366570 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4a814d730 con 0x7fb4a80b4710 2024-03-29T11:19:48.594 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.614+0000 7fb4a4ff9640 1 -- 172.21.15.170:0/3002366570 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb49c004080 con 0x7fb4a80b4710 2024-03-29T11:19:48.594 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.614+0000 7fb4a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:48.594 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.614+0000 7fb4a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:48.594 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.614+0000 7fb4a4ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:48.594 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:48.594 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:48.595 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:48.595 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:48.595 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:48.595 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:48.595 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:48.595 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:48.595 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:48.595 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:48.595 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.614+0000 7fb4a4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:48.595 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.614+0000 7fb4a4ff9640 1 -- 172.21.15.170:0/3002366570 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1698767397 0 0) 0x7fb4a814d730 con 0x7fb4a80b4710 2024-03-29T11:19:48.595 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.614+0000 7fb4a4ff9640 1 -- 172.21.15.170:0/3002366570 >> v1:172.21.15.184:6789/0 conn(0x7fb4a813f390 legacy=0x7fb4a814d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:48.595 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.614+0000 7fb4a4ff9640 1 -- 172.21.15.170:0/3002366570 >> v1:172.21.15.170:6789/0 conn(0x7fb4a80b3970 legacy=0x7fb4a806e910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:48.595 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.615+0000 7fb4a4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:19:48.595 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.615+0000 7fb4a4ff9640 1 monclient: found mon.c 2024-03-29T11:19:48.595 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.615+0000 7fb4a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:48.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.615+0000 7fb4a4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:48.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.615+0000 7fb4a4ff9640 1 -- 172.21.15.170:0/3002366570 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a814bc80 con 0x7fb4a80b4710 2024-03-29T11:19:48.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.615+0000 7fb4a4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:19:48.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.615+0000 7fb4a4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:19:48.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.615+0000 7fb4a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:48.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.615+0000 7fb4ae546640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:19:48.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.615+0000 7fb4ae546640 10 monclient: _renew_subs 2024-03-29T11:19:48.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.615+0000 7fb4ae546640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:48.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.615+0000 7fb4ae546640 1 -- 172.21.15.170:0/3002366570 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a81e70c0 con 0x7fb4a80b4710 2024-03-29T11:19:48.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.615+0000 7fb4a4ff9640 1 -- 172.21.15.170:0/3002366570 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb49c0042e0 con 0x7fb4a80b4710 2024-03-29T11:19:48.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.615+0000 7fb4a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:48.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.615+0000 7fb4ae546640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:48.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.615+0000 7fb4ae546640 10 monclient: _renew_subs 2024-03-29T11:19:48.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.615+0000 7fb4a67fc640 4 set_mon_vals no callback set 2024-03-29T11:19:48.596 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.616+0000 7fb4ae546640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:48.597 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.616+0000 7fb4ae546640 1 -- 172.21.15.170:0/3002366570 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a81e7430 con 0x7fb4a80b4710 2024-03-29T11:19:48.597 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.616+0000 7fb4ae546640 1 librados: init done 2024-03-29T11:19:48.597 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.616+0000 7fb4a4ff9640 1 -- 172.21.15.170:0/3002366570 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fb49c001320 con 0x7fb4a80b4710 2024-03-29T11:19:48.597 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.616+0000 7fb4a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:48.597 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.616+0000 7fb4a4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:48.597 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.616+0000 7fb4a4ff9640 10 monclient: dump: 2024-03-29T11:19:48.597 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:48.597 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:48.597 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:48.597 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:48.597 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:48.597 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:48.597 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:48.598 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:48.598 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:48.598 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:48.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.618+0000 7fb4ae546640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:48.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.618+0000 7fb4ae546640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:48.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.618+0000 7fb4ae546640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:48.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.618+0000 7fb4a4ff9640 1 -- 172.21.15.170:0/3002366570 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fb49c001500 con 0x7fb4a80b4710 2024-03-29T11:19:48.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.618+0000 7fb4ae546640 1 -- 172.21.15.170:0/3002366570 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a814d910 con 0x7fb4a80b4710 2024-03-29T11:19:48.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.618+0000 7fb4a4ff9640 1 -- 172.21.15.170:0/3002366570 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fb49c084c70 con 0x7fb4a80b4710 2024-03-29T11:19:48.599 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.618+0000 7fb4a4ff9640 10 client.4559.objecter ms_dispatch 0x7fb4a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:48.600 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.618+0000 7fb4a4ff9640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:48.600 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.618+0000 7fb4a4ff9640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:48.600 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.618+0000 7fb4a4ff9640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-29T11:19:48.600 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.619+0000 7fb4a7fff640 10 monclient: get_auth_request con 0x7fb48c06dbf0 auth_method 0 2024-03-29T11:19:48.600 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.619+0000 7fb4a4ff9640 10 client.4559.objecter ms_handle_connect 0x7fb48c06dbf0 2024-03-29T11:19:48.602 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.626+0000 7fb4a4ff9640 1 -- 172.21.15.170:0/3002366570 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb49c050380 con 0x7fb4a80b4710 2024-03-29T11:19:48.602 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.626+0000 7fb4a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:48.602 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.626+0000 7fb4a4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:48.918 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.942+0000 7fb4ae546640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:48.918 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.942+0000 7fb4ae546640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:48.919 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.942+0000 7fb4ae546640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:48.919 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.942+0000 7fb4ae546640 1 -- 172.21.15.170:0/3002366570 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4a80688a0 con 0x7fb4a80b4710 2024-03-29T11:19:48.919 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.943+0000 7fb4a4ff9640 1 -- 172.21.15.170:0/3002366570 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fb49c052480 con 0x7fb4a80b4710 2024-03-29T11:19:48.919 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.943+0000 7fb4a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:48.919 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.943+0000 7fb4a4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:48.920 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:19:48.920 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:48.924 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.947+0000 7fb4ae546640 10 librados: watch_flush enter 2024-03-29T11:19:48.924 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.947+0000 7fb4ae546640 10 librados: watch_flush exit 2024-03-29T11:19:48.924 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.947+0000 7fb4ae546640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-29T11:19:48.924 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.947+0000 7fb4ae546640 10 client.4559.objecter successfully canceled tick 2024-03-29T11:19:48.924 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.947+0000 7fb4ae546640 1 -- 172.21.15.170:0/3002366570 >> v1:172.21.15.170:6816/3298994444 conn(0x7fb48c06dbf0 legacy=0x7fb48c0700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:48.924 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.947+0000 7fb4ae546640 10 monclient: shutdown 2024-03-29T11:19:48.924 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.947+0000 7fb4ae546640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:48.924 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.947+0000 7fb4ae546640 1 -- 172.21.15.170:0/3002366570 >> v1:172.21.15.170:6790/0 conn(0x7fb4a80b4710 legacy=0x7fb4a81425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:48.924 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.947+0000 7fb4ae546640 1 -- 172.21.15.170:0/3002366570 shutdown_connections 2024-03-29T11:19:48.924 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.947+0000 7fb4ae546640 1 -- 172.21.15.170:0/3002366570 >> 172.21.15.170:0/3002366570 conn(0x7fb4a8066a00 msgr2=0x7fb4a80b1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:48.924 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.947+0000 7fb4ae546640 1 -- 172.21.15.170:0/3002366570 shutdown_connections 2024-03-29T11:19:48.924 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.947+0000 7fb4ae546640 1 -- 172.21.15.170:0/3002366570 wait complete. 2024-03-29T11:19:48.924 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:48.947+0000 7fb4ae546640 1 librados: shutdown 2024-03-29T11:19:48.939 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:55.746 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:55.880 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.903+0000 7fcfa436b640 10 monclient: get_monmap_and_config 2024-03-29T11:19:55.880 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.903+0000 7fcfa436b640 10 monclient: build_initial_monmap 2024-03-29T11:19:55.880 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.903+0000 7fcfa436b640 10 monclient: monmap: 2024-03-29T11:19:55.880 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:19:55.880 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:55.880 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:19:55.904260+0000 2024-03-29T11:19:55.880 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:19:55.904260+0000 2024-03-29T11:19:55.880 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:55.880 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:55.880 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:19:55.881 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:19:55.881 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:19:55.881 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:55.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.904+0000 7fcfa436b640 1 Processor -- start 2024-03-29T11:19:55.881 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.904+0000 7fcfa436b640 1 -- start start 2024-03-29T11:19:55.882 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.904+0000 7fcfa436b640 10 monclient: init 2024-03-29T11:19:55.882 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.905+0000 7fcfa436b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:55.882 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.905+0000 7fcfa436b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:55.882 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.905+0000 7fcfa436b640 10 monclient(hunting): picked mon.noname-c con 0x7fcf9c1422b0 addr v1:172.21.15.184:6789/0 2024-03-29T11:19:55.882 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.905+0000 7fcfa436b640 10 monclient(hunting): picked mon.noname-b con 0x7fcf9c142690 addr v1:172.21.15.170:6790/0 2024-03-29T11:19:55.883 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.906+0000 7fcfa2107640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fcf9c1422b0 0x7fcf9c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:57488/0 (socket says 172.21.15.170:57488) 2024-03-29T11:19:55.883 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.906+0000 7fcfa436b640 10 monclient(hunting): picked mon.noname-a con 0x7fcf9c0b17d0 addr v1:172.21.15.170:6789/0 2024-03-29T11:19:55.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.906+0000 7fcfa436b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9c142c80 con 0x7fcf9c0b17d0 2024-03-29T11:19:55.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.906+0000 7fcfa436b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9c0aae40 con 0x7fcf9c1422b0 2024-03-29T11:19:55.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.906+0000 7fcfa436b640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9c13fbd0 con 0x7fcf9c142690 2024-03-29T11:19:55.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.906+0000 7fcfa436b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:55.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.906+0000 7fcfa436b640 10 monclient(hunting): authenticate will time out at 1377.217282s 2024-03-29T11:19:55.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.906+0000 7fcfa1906640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fcf9c142690 0x7fcf9c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:36802/0 (socket says 172.21.15.170:36802) 2024-03-29T11:19:55.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.906+0000 7fcfa1906640 1 -- 172.21.15.170:0/1981066517 learned_addr learned my addr 172.21.15.170:0/1981066517 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:19:55.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.907+0000 7fcfa1105640 1 -- 172.21.15.170:0/1981066517 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240190980 0 0) 0x7fcf9c0aae40 con 0x7fcf9c1422b0 2024-03-29T11:19:55.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.907+0000 7fcfa1105640 10 monclient(hunting): _negotiate 2024-03-29T11:19:55.907 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.907+0000 7fcfa1105640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:55.907 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.907+0000 7fcfa1105640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:55.907 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.907+0000 7fcfa1105640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:19:55.907 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.907+0000 7fcfa1105640 1 -- 172.21.15.170:0/1981066517 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf88003520 con 0x7fcf9c1422b0 2024-03-29T11:19:55.907 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.907+0000 7fcfa1105640 1 -- 172.21.15.170:0/1981066517 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384055361 0 0) 0x7fcf9c13fbd0 con 0x7fcf9c142690 2024-03-29T11:19:55.907 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.907+0000 7fcfa1105640 10 monclient(hunting): _negotiate 2024-03-29T11:19:55.907 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.907+0000 7fcfa1105640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:55.907 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.907+0000 7fcfa1105640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:55.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.907+0000 7fcfa1105640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:19:55.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.907+0000 7fcfa1105640 1 -- 172.21.15.170:0/1981066517 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf9c0aae40 con 0x7fcf9c142690 2024-03-29T11:19:55.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.907+0000 7fcfa1105640 1 -- 172.21.15.170:0/1981066517 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2619290792 0 0) 0x7fcf9c0aae40 con 0x7fcf9c142690 2024-03-29T11:19:55.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.907+0000 7fcfa1105640 1 -- 172.21.15.170:0/1981066517 >> v1:172.21.15.184:6789/0 conn(0x7fcf9c1422b0 legacy=0x7fcf9c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:55.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.907+0000 7fcfa1105640 1 -- 172.21.15.170:0/1981066517 >> v1:172.21.15.170:6789/0 conn(0x7fcf9c0b17d0 legacy=0x7fcf9c0b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:55.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa1105640 10 monclient: _finish_hunting 0 2024-03-29T11:19:55.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa1105640 1 monclient: found mon.noname-b 2024-03-29T11:19:55.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa1105640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:55.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa1105640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:19:55.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa1105640 1 -- 172.21.15.170:0/1981066517 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c151940 con 0x7fcf9c142690 2024-03-29T11:19:55.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa1105640 10 monclient: _finish_auth 0 2024-03-29T11:19:55.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa1105640 10 monclient: _check_auth_tickets 2024-03-29T11:19:55.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa1105640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:55.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa1105640 1 -- 172.21.15.170:0/1981066517 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fcf8c001670 con 0x7fcf9c142690 2024-03-29T11:19:55.909 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa436b640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:19:55.909 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa436b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:55.909 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa1105640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:55.909 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa1105640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:55.909 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa1105640 10 monclient: dump: 2024-03-29T11:19:55.909 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:55.909 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:55.909 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:55.909 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:55.909 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:55.909 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:55.909 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:55.909 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:55.909 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:55.909 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:55.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa436b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:55.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa1105640 1 -- 172.21.15.170:0/1981066517 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf8c0017d0 con 0x7fcf9c142690 2024-03-29T11:19:55.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa1105640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:55.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa1105640 1 -- 172.21.15.170:0/1981066517 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fcf8c001ba0 con 0x7fcf9c142690 2024-03-29T11:19:55.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa436b640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:55.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa436b640 4 set_mon_vals no callback set 2024-03-29T11:19:55.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa1105640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:55.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa1105640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:55.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.908+0000 7fcfa1105640 10 monclient: dump: 2024-03-29T11:19:55.910 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:55.910 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:55.910 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:55.910 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:55.910 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:55.911 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:55.911 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:55.911 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:55.911 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:55.911 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:55.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.909+0000 7fcfa436b640 10 monclient: shutdown 2024-03-29T11:19:55.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.909+0000 7fcfa436b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:55.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.909+0000 7fcfa436b640 1 -- 172.21.15.170:0/1981066517 >> v1:172.21.15.170:6790/0 conn(0x7fcf9c142690 legacy=0x7fcf9c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:55.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.909+0000 7fcfa436b640 1 -- 172.21.15.170:0/1981066517 shutdown_connections 2024-03-29T11:19:55.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.909+0000 7fcfa436b640 1 -- 172.21.15.170:0/1981066517 >> 172.21.15.170:0/1981066517 conn(0x7fcf9c066a00 msgr2=0x7fcf9c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:55.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.909+0000 7fcfa436b640 1 -- 172.21.15.170:0/1981066517 shutdown_connections 2024-03-29T11:19:55.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.909+0000 7fcfa436b640 1 -- 172.21.15.170:0/1981066517 wait complete. 2024-03-29T11:19:55.914 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.932+0000 7fcfa436b640 10 monclient: build_initial_monmap 2024-03-29T11:19:55.914 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.932+0000 7fcfa436b640 10 monclient: monmap: 2024-03-29T11:19:55.914 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:19:55.914 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:55.914 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:19:55.914 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:19:55.914 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:55.914 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:55.914 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:19:55.914 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:19:55.914 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:19:55.914 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:55.914 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.933+0000 7fcfa436b640 1 librados: starting msgr at 2024-03-29T11:19:55.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.933+0000 7fcfa436b640 1 librados: starting objecter 2024-03-29T11:19:55.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.933+0000 7fcfa436b640 1 Processor -- start 2024-03-29T11:19:55.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.934+0000 7fcfa436b640 1 -- start start 2024-03-29T11:19:55.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.934+0000 7fcfa436b640 1 librados: setting wanted keys 2024-03-29T11:19:55.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.934+0000 7fcfa436b640 1 librados: calling monclient init 2024-03-29T11:19:55.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.934+0000 7fcfa436b640 10 monclient: init 2024-03-29T11:19:55.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.934+0000 7fcfa436b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:55.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.934+0000 7fcfa436b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:55.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.934+0000 7fcfa436b640 10 monclient(hunting): picked mon.noname-b con 0x7fcf9c0b17d0 addr v1:172.21.15.184:6789/0 2024-03-29T11:19:55.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.935+0000 7fcfa436b640 10 monclient(hunting): picked mon.noname-c con 0x7fcf9c1422b0 addr v1:172.21.15.170:6790/0 2024-03-29T11:19:55.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.935+0000 7fcfa1906640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fcf9c1422b0 0x7fcf9c12e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:39168/0 (socket says 172.21.15.170:39168) 2024-03-29T11:19:55.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.935+0000 7fcfa1906640 1 -- 172.21.15.170:0/765291503 learned_addr learned my addr 172.21.15.170:0/765291503 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:19:55.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.935+0000 7fcfa436b640 10 monclient(hunting): picked mon.noname-a con 0x7fcf9c142690 addr v1:172.21.15.170:6789/0 2024-03-29T11:19:55.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.935+0000 7fcfa436b640 1 -- 172.21.15.170:0/765291503 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9c0acb30 con 0x7fcf9c142690 2024-03-29T11:19:55.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.935+0000 7fcfa436b640 1 -- 172.21.15.170:0/765291503 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9c0ace10 con 0x7fcf9c0b17d0 2024-03-29T11:19:55.916 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.935+0000 7fcfa436b640 1 -- 172.21.15.170:0/765291503 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9c0ab5e0 con 0x7fcf9c1422b0 2024-03-29T11:19:55.916 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.935+0000 7fcf92ffd640 10 client.?.objecter ms_handle_connect 0x7fcf9c1422b0 2024-03-29T11:19:55.916 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.935+0000 7fcf92ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:55.916 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.935+0000 7fcfa436b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:55.916 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.935+0000 7fcfa436b640 10 monclient(hunting): authenticate will time out at 1377.246366s 2024-03-29T11:19:55.916 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.935+0000 7fcf92ffd640 1 -- 172.21.15.170:0/765291503 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967102342 0 0) 0x7fcf9c0ab5e0 con 0x7fcf9c1422b0 2024-03-29T11:19:55.916 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.935+0000 7fcf92ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:55.916 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.935+0000 7fcf92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:55.916 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.935+0000 7fcf92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:55.916 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.935+0000 7fcf92ffd640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:19:55.916 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.935+0000 7fcf92ffd640 1 -- 172.21.15.170:0/765291503 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf80004eb0 con 0x7fcf9c1422b0 2024-03-29T11:19:55.916 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.936+0000 7fcf92ffd640 1 -- 172.21.15.170:0/765291503 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1227662244 0 0) 0x7fcf80004eb0 con 0x7fcf9c1422b0 2024-03-29T11:19:55.916 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.936+0000 7fcf92ffd640 1 -- 172.21.15.170:0/765291503 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf9c0ab5e0 con 0x7fcf9c1422b0 2024-03-29T11:19:55.917 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.936+0000 7fcf92ffd640 1 -- 172.21.15.170:0/765291503 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fcf8c001ba0 con 0x7fcf9c1422b0 2024-03-29T11:19:55.917 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.936+0000 7fcf92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:55.917 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.936+0000 7fcf92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:55.917 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.936+0000 7fcf92ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:55.917 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:55.917 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:55.917 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:55.917 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:55.917 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:55.917 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:55.917 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:55.917 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:55.917 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:55.917 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:55.917 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.936+0000 7fcf92ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:55.918 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.936+0000 7fcf92ffd640 1 -- 172.21.15.170:0/765291503 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 874704538 0 0) 0x7fcf9c0ab5e0 con 0x7fcf9c1422b0 2024-03-29T11:19:55.918 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.936+0000 7fcf92ffd640 1 -- 172.21.15.170:0/765291503 >> v1:172.21.15.184:6789/0 conn(0x7fcf9c0b17d0 legacy=0x7fcf9c150730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:55.918 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.936+0000 7fcf92ffd640 1 -- 172.21.15.170:0/765291503 >> v1:172.21.15.170:6789/0 conn(0x7fcf9c142690 legacy=0x7fcf9c14d880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:55.920 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcf92ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:55.920 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcf92ffd640 1 monclient: found mon.c 2024-03-29T11:19:55.920 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcf92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:55.920 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcf92ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:55.920 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcf92ffd640 1 -- 172.21.15.170:0/765291503 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c0abb00 con 0x7fcf9c1422b0 2024-03-29T11:19:55.920 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcf92ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:55.920 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcf92ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:55.920 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcf92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:55.920 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcfa436b640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:19:55.921 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcfa436b640 10 monclient: _renew_subs 2024-03-29T11:19:55.921 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcfa436b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:55.921 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcfa436b640 1 -- 172.21.15.170:0/765291503 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9c12e900 con 0x7fcf9c1422b0 2024-03-29T11:19:55.921 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcfa436b640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:55.921 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcf92ffd640 1 -- 172.21.15.170:0/765291503 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf8c001d00 con 0x7fcf9c1422b0 2024-03-29T11:19:55.921 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcf92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:55.921 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcf92ffd640 1 -- 172.21.15.170:0/765291503 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fcf8c004d30 con 0x7fcf9c1422b0 2024-03-29T11:19:55.921 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcf92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:55.921 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcf92ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:55.921 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcf92ffd640 10 monclient: dump: 2024-03-29T11:19:55.921 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:19:55.921 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:19:55.921 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:55.921 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:19:55.921 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:55.921 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:19:55.922 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:19:55.922 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:19:55.922 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:19:55.922 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:19:55.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcfa436b640 10 monclient: _renew_subs 2024-03-29T11:19:55.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcfa436b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:55.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcfa436b640 1 -- 172.21.15.170:0/765291503 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9c131c50 con 0x7fcf9c1422b0 2024-03-29T11:19:55.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.938+0000 7fcfa436b640 1 librados: init done 2024-03-29T11:19:55.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.939+0000 7fcfa0904640 4 set_mon_vals no callback set 2024-03-29T11:19:55.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.940+0000 7fcf92ffd640 1 -- 172.21.15.170:0/765291503 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fcf8c01ad00 con 0x7fcf9c1422b0 2024-03-29T11:19:55.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.941+0000 7fcfa436b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:55.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.941+0000 7fcfa436b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:55.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.941+0000 7fcfa436b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:55.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.941+0000 7fcfa436b640 1 -- 172.21.15.170:0/765291503 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf84005080 con 0x7fcf9c1422b0 2024-03-29T11:19:55.923 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.941+0000 7fcf92ffd640 1 -- 172.21.15.170:0/765291503 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fcf8c084c20 con 0x7fcf9c1422b0 2024-03-29T11:19:55.923 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.941+0000 7fcf92ffd640 10 client.4565.objecter ms_dispatch 0x7fcf9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:55.923 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.941+0000 7fcf92ffd640 3 client.4565.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:55.923 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.941+0000 7fcf92ffd640 3 client.4565.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:55.923 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.941+0000 7fcf92ffd640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-29T11:19:55.923 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.941+0000 7fcfa2107640 10 monclient: get_auth_request con 0x7fcf8006dd00 auth_method 0 2024-03-29T11:19:55.923 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.941+0000 7fcf92ffd640 10 client.4565.objecter ms_handle_connect 0x7fcf8006dd00 2024-03-29T11:19:55.924 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.948+0000 7fcf92ffd640 1 -- 172.21.15.170:0/765291503 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcf8c050500 con 0x7fcf9c1422b0 2024-03-29T11:19:55.924 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.948+0000 7fcf92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:55.924 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:55.948+0000 7fcf92ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:56.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.253+0000 7fcfa436b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:56.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.253+0000 7fcfa436b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:56.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.253+0000 7fcfa436b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:19:56.229 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.253+0000 7fcfa436b640 1 -- 172.21.15.170:0/765291503 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf84005330 con 0x7fcf9c1422b0 2024-03-29T11:19:56.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.253+0000 7fcf92ffd640 1 -- 172.21.15.170:0/765291503 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fcf8c052430 con 0x7fcf9c1422b0 2024-03-29T11:19:56.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.254+0000 7fcf92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:56.230 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.254+0000 7fcf92ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:56.230 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:19:56.231 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:56.233 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.256+0000 7fcfa436b640 10 librados: watch_flush enter 2024-03-29T11:19:56.233 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.256+0000 7fcfa436b640 10 librados: watch_flush exit 2024-03-29T11:19:56.233 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.256+0000 7fcfa436b640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-29T11:19:56.233 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.256+0000 7fcfa436b640 10 client.4565.objecter successfully canceled tick 2024-03-29T11:19:56.234 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.257+0000 7fcfa436b640 1 -- 172.21.15.170:0/765291503 >> v1:172.21.15.170:6816/3298994444 conn(0x7fcf8006dd00 legacy=0x7fcf800701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.234 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.257+0000 7fcfa436b640 10 monclient: shutdown 2024-03-29T11:19:56.234 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.257+0000 7fcfa436b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:56.234 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.257+0000 7fcfa436b640 1 -- 172.21.15.170:0/765291503 >> v1:172.21.15.170:6790/0 conn(0x7fcf9c1422b0 legacy=0x7fcf9c12e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.234 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.257+0000 7fcfa436b640 1 -- 172.21.15.170:0/765291503 shutdown_connections 2024-03-29T11:19:56.234 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.257+0000 7fcfa436b640 1 -- 172.21.15.170:0/765291503 >> 172.21.15.170:0/765291503 conn(0x7fcf9c066a00 msgr2=0x7fcf9c13c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:56.234 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.257+0000 7fcfa436b640 1 -- 172.21.15.170:0/765291503 shutdown_connections 2024-03-29T11:19:56.234 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.257+0000 7fcfa436b640 1 -- 172.21.15.170:0/765291503 wait complete. 2024-03-29T11:19:56.234 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:19:56.257+0000 7fcfa436b640 1 librados: shutdown 2024-03-29T11:19:56.246 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:03.052 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:03.197 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.221+0000 7f92fba8f640 10 monclient: get_monmap_and_config 2024-03-29T11:20:03.197 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.221+0000 7f92fba8f640 10 monclient: build_initial_monmap 2024-03-29T11:20:03.197 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.221+0000 7f92fba8f640 10 monclient: monmap: 2024-03-29T11:20:03.198 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:20:03.198 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:03.198 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:20:03.222116+0000 2024-03-29T11:20:03.198 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:20:03.222116+0000 2024-03-29T11:20:03.198 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:03.198 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:03.198 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:20:03.198 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:20:03.198 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:20:03.198 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:03.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.222+0000 7f92fba8f640 1 Processor -- start 2024-03-29T11:20:03.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.223+0000 7f92fba8f640 1 -- start start 2024-03-29T11:20:03.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.223+0000 7f92fba8f640 10 monclient: init 2024-03-29T11:20:03.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.224+0000 7f92fba8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:03.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.224+0000 7f92fba8f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:03.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.224+0000 7f92fba8f640 10 monclient(hunting): picked mon.noname-c con 0x7f92f41422b0 addr v1:172.21.15.184:6789/0 2024-03-29T11:20:03.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.224+0000 7f92fba8f640 10 monclient(hunting): picked mon.noname-b con 0x7f92f4142690 addr v1:172.21.15.170:6790/0 2024-03-29T11:20:03.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.224+0000 7f92fba8f640 10 monclient(hunting): picked mon.noname-a con 0x7f92f40b17d0 addr v1:172.21.15.170:6789/0 2024-03-29T11:20:03.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.224+0000 7f92fba8f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f4142c80 con 0x7f92f40b17d0 2024-03-29T11:20:03.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.224+0000 7f92fba8f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f40aae40 con 0x7f92f41422b0 2024-03-29T11:20:03.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.224+0000 7f92fba8f640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f413fbd0 con 0x7f92f4142690 2024-03-29T11:20:03.202 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.224+0000 7f92fba8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:03.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.224+0000 7f92fba8f640 10 monclient(hunting): authenticate will time out at 1384.535228s 2024-03-29T11:20:03.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.224+0000 7f92f902a640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f92f4142690 0x7f92f40b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:47768/0 (socket says 172.21.15.170:47768) 2024-03-29T11:20:03.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.224+0000 7f92f902a640 1 -- 172.21.15.170:0/2881218108 learned_addr learned my addr 172.21.15.170:0/2881218108 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:20:03.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.226+0000 7f92f8829640 1 -- 172.21.15.170:0/2881218108 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927451568 0 0) 0x7f92f413fbd0 con 0x7f92f4142690 2024-03-29T11:20:03.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.226+0000 7f92f8829640 10 monclient(hunting): _negotiate 2024-03-29T11:20:03.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.226+0000 7f92f8829640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:03.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.226+0000 7f92f8829640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:03.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.226+0000 7f92f8829640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:20:03.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.226+0000 7f92f8829640 1 -- 172.21.15.170:0/2881218108 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e0003520 con 0x7f92f4142690 2024-03-29T11:20:03.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 1 -- 172.21.15.170:0/2881218108 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53485485 0 0) 0x7f92f4142c80 con 0x7f92f40b17d0 2024-03-29T11:20:03.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 10 monclient(hunting): _negotiate 2024-03-29T11:20:03.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:03.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:03.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:20:03.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 1 -- 172.21.15.170:0/2881218108 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f413fbd0 con 0x7f92f40b17d0 2024-03-29T11:20:03.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 1 -- 172.21.15.170:0/2881218108 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1832531461 0 0) 0x7f92e0003520 con 0x7f92f4142690 2024-03-29T11:20:03.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 1 -- 172.21.15.170:0/2881218108 >> v1:172.21.15.184:6789/0 conn(0x7f92f41422b0 legacy=0x7f92f413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:03.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 1 -- 172.21.15.170:0/2881218108 >> v1:172.21.15.170:6789/0 conn(0x7f92f40b17d0 legacy=0x7f92f40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:03.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 10 monclient: _finish_hunting 0 2024-03-29T11:20:03.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 1 monclient: found mon.noname-b 2024-03-29T11:20:03.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:03.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:20:03.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 1 -- 172.21.15.170:0/2881218108 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f4151940 con 0x7f92f4142690 2024-03-29T11:20:03.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 10 monclient: _finish_auth 0 2024-03-29T11:20:03.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 10 monclient: _check_auth_tickets 2024-03-29T11:20:03.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:03.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92fba8f640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:20:03.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92fba8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:03.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 1 -- 172.21.15.170:0/2881218108 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f92ec001990 con 0x7f92f4142690 2024-03-29T11:20:03.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:03.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:03.205 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92f8829640 10 monclient: dump: 2024-03-29T11:20:03.205 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:03.205 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:03.205 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:03.205 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:03.205 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:03.206 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:03.206 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:03.206 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:03.206 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:03.206 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:03.206 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.227+0000 7f92fba8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:03.206 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.228+0000 7f92f8829640 1 -- 172.21.15.170:0/2881218108 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92ec001bb0 con 0x7f92f4142690 2024-03-29T11:20:03.206 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.228+0000 7f92f8829640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:03.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.228+0000 7f92fba8f640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:03.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.228+0000 7f92fba8f640 4 set_mon_vals no callback set 2024-03-29T11:20:03.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.228+0000 7f92f8829640 1 -- 172.21.15.170:0/2881218108 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f92ec001f40 con 0x7f92f4142690 2024-03-29T11:20:03.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.228+0000 7f92f8829640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:03.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.228+0000 7f92f8829640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:03.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.228+0000 7f92f8829640 10 monclient: dump: 2024-03-29T11:20:03.207 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:03.207 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:03.207 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:03.207 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:03.207 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:03.207 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:03.207 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:03.207 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:03.208 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:03.208 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:03.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.229+0000 7f92fba8f640 10 monclient: shutdown 2024-03-29T11:20:03.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.229+0000 7f92fba8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:03.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.229+0000 7f92fba8f640 1 -- 172.21.15.170:0/2881218108 >> v1:172.21.15.170:6790/0 conn(0x7f92f4142690 legacy=0x7f92f40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:03.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.229+0000 7f92fba8f640 1 -- 172.21.15.170:0/2881218108 shutdown_connections 2024-03-29T11:20:03.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.229+0000 7f92fba8f640 1 -- 172.21.15.170:0/2881218108 >> 172.21.15.170:0/2881218108 conn(0x7f92f4066a00 msgr2=0x7f92f4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:03.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.230+0000 7f92fba8f640 1 -- 172.21.15.170:0/2881218108 shutdown_connections 2024-03-29T11:20:03.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.230+0000 7f92fba8f640 1 -- 172.21.15.170:0/2881218108 wait complete. 2024-03-29T11:20:03.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.230+0000 7f92fba8f640 10 monclient: build_initial_monmap 2024-03-29T11:20:03.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.230+0000 7f92fba8f640 10 monclient: monmap: 2024-03-29T11:20:03.208 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:20:03.208 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:03.208 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:20:03.208 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:20:03.209 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:03.209 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:03.209 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:20:03.209 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:20:03.209 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:20:03.209 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:03.211 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.231+0000 7f92fba8f640 1 librados: starting msgr at 2024-03-29T11:20:03.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.231+0000 7f92fba8f640 1 librados: starting objecter 2024-03-29T11:20:03.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.231+0000 7f92fba8f640 1 Processor -- start 2024-03-29T11:20:03.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.231+0000 7f92fba8f640 1 -- start start 2024-03-29T11:20:03.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.231+0000 7f92fba8f640 1 librados: setting wanted keys 2024-03-29T11:20:03.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.231+0000 7f92fba8f640 1 librados: calling monclient init 2024-03-29T11:20:03.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.231+0000 7f92fba8f640 10 monclient: init 2024-03-29T11:20:03.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.232+0000 7f92fba8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:03.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.232+0000 7f92fba8f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:03.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.232+0000 7f92fba8f640 10 monclient(hunting): picked mon.noname-b con 0x7f92f40b17d0 addr v1:172.21.15.184:6789/0 2024-03-29T11:20:03.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.232+0000 7f92fba8f640 10 monclient(hunting): picked mon.noname-c con 0x7f92f41422b0 addr v1:172.21.15.170:6790/0 2024-03-29T11:20:03.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.232+0000 7f92fba8f640 10 monclient(hunting): picked mon.noname-a con 0x7f92f414bb70 addr v1:172.21.15.170:6789/0 2024-03-29T11:20:03.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.232+0000 7f92fba8f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f414bfe0 con 0x7f92f414bb70 2024-03-29T11:20:03.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.232+0000 7f92fba8f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f40b6d50 con 0x7f92f40b17d0 2024-03-29T11:20:03.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.232+0000 7f92fba8f640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f40b7030 con 0x7f92f41422b0 2024-03-29T11:20:03.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.232+0000 7f92fba8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:03.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.232+0000 7f92fba8f640 10 monclient(hunting): authenticate will time out at 1384.543462s 2024-03-29T11:20:03.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.232+0000 7f92f902a640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f92f41422b0 0x7f92f40b5bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:48736/0 (socket says 172.21.15.170:48736) 2024-03-29T11:20:03.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.232+0000 7f92f902a640 1 -- 172.21.15.170:0/1117264430 learned_addr learned my addr 172.21.15.170:0/1117264430 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:20:03.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.234+0000 7f92ea7fc640 10 client.?.objecter ms_handle_connect 0x7f92f41422b0 2024-03-29T11:20:03.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.234+0000 7f92ea7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:03.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.234+0000 7f92ea7fc640 1 -- 172.21.15.170:0/1117264430 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2153978896 0 0) 0x7f92f40b7030 con 0x7f92f41422b0 2024-03-29T11:20:03.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.234+0000 7f92ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:03.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.234+0000 7f92ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:03.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.234+0000 7f92ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:03.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.234+0000 7f92ea7fc640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:20:03.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.234+0000 7f92ea7fc640 1 -- 172.21.15.170:0/1117264430 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d8004eb0 con 0x7f92f41422b0 2024-03-29T11:20:03.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.234+0000 7f92ea7fc640 1 -- 172.21.15.170:0/1117264430 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2012152505 0 0) 0x7f92d8004eb0 con 0x7f92f41422b0 2024-03-29T11:20:03.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.234+0000 7f92ea7fc640 1 -- 172.21.15.170:0/1117264430 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92f40b7030 con 0x7f92f41422b0 2024-03-29T11:20:03.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.234+0000 7f92ea7fc640 1 -- 172.21.15.170:0/1117264430 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f92ec0018d0 con 0x7f92f41422b0 2024-03-29T11:20:03.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.234+0000 7f92ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:03.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.234+0000 7f92ea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:03.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.234+0000 7f92ea7fc640 10 monclient(hunting): dump: 2024-03-29T11:20:03.214 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:03.214 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:03.214 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:03.214 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:03.214 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:03.214 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:03.214 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:03.214 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:03.214 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:03.215 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:03.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.234+0000 7f92ea7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:03.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.235+0000 7f92ea7fc640 1 -- 172.21.15.170:0/1117264430 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3825698269 0 0) 0x7f92f40b7030 con 0x7f92f41422b0 2024-03-29T11:20:03.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.235+0000 7f92ea7fc640 1 -- 172.21.15.170:0/1117264430 >> v1:172.21.15.184:6789/0 conn(0x7f92f40b17d0 legacy=0x7f92f4140f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:03.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.236+0000 7f92ea7fc640 1 -- 172.21.15.170:0/1117264430 >> v1:172.21.15.170:6789/0 conn(0x7f92f414bb70 legacy=0x7f92f41e68b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:03.218 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.237+0000 7f92ea7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:03.218 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.237+0000 7f92ea7fc640 1 monclient: found mon.c 2024-03-29T11:20:03.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.237+0000 7f92ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:03.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.237+0000 7f92ea7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:20:03.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.237+0000 7f92ea7fc640 1 -- 172.21.15.170:0/1117264430 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f414e420 con 0x7f92f41422b0 2024-03-29T11:20:03.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.237+0000 7f92ea7fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:03.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.237+0000 7f92ea7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:03.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.237+0000 7f92ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:03.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.237+0000 7f92fba8f640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:20:03.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.237+0000 7f92fba8f640 10 monclient: _renew_subs 2024-03-29T11:20:03.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.237+0000 7f92fba8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:20:03.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.237+0000 7f92fba8f640 1 -- 172.21.15.170:0/1117264430 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f4150ea0 con 0x7f92f41422b0 2024-03-29T11:20:03.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.237+0000 7f92fba8f640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:03.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.237+0000 7f92fba8f640 10 monclient: _renew_subs 2024-03-29T11:20:03.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.237+0000 7f92fba8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:20:03.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.237+0000 7f92fba8f640 1 -- 172.21.15.170:0/1117264430 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f41e81b0 con 0x7f92f41422b0 2024-03-29T11:20:03.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.237+0000 7f92fba8f640 1 librados: init done 2024-03-29T11:20:03.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.239+0000 7f92fba8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:03.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.239+0000 7f92fba8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:03.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.239+0000 7f92fba8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:20:03.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.239+0000 7f92fba8f640 1 -- 172.21.15.170:0/1117264430 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f40b7550 con 0x7f92f41422b0 2024-03-29T11:20:03.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.239+0000 7f92ea7fc640 1 -- 172.21.15.170:0/1117264430 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92ec004e10 con 0x7f92f41422b0 2024-03-29T11:20:03.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.239+0000 7f92ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:03.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.241+0000 7f92ea7fc640 1 -- 172.21.15.170:0/1117264430 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f92ec0051e0 con 0x7f92f41422b0 2024-03-29T11:20:03.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.241+0000 7f92ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:03.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.241+0000 7f92ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:03.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.241+0000 7f92ea7fc640 10 monclient: dump: 2024-03-29T11:20:03.220 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:03.220 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:03.220 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:03.220 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:03.221 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:03.221 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:03.221 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:03.221 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:03.221 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:03.221 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:03.221 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.241+0000 7f92ea7fc640 1 -- 172.21.15.170:0/1117264430 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f92ec005400 con 0x7f92f41422b0 2024-03-29T11:20:03.221 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.241+0000 7f92ebfff640 4 set_mon_vals no callback set 2024-03-29T11:20:03.221 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.241+0000 7f92ea7fc640 1 -- 172.21.15.170:0/1117264430 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f92ec084e20 con 0x7f92f41422b0 2024-03-29T11:20:03.221 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.241+0000 7f92ea7fc640 10 client.4571.objecter ms_dispatch 0x7f92f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:03.221 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.241+0000 7f92ea7fc640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:03.221 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.241+0000 7f92ea7fc640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:03.221 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.241+0000 7f92ea7fc640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-29T11:20:03.221 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.242+0000 7f92f982b640 10 monclient: get_auth_request con 0x7f92d806dcd0 auth_method 0 2024-03-29T11:20:03.221 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.242+0000 7f92ea7fc640 10 client.4571.objecter ms_handle_connect 0x7f92d806dcd0 2024-03-29T11:20:03.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.249+0000 7f92ea7fc640 1 -- 172.21.15.170:0/1117264430 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f92ec050780 con 0x7f92f41422b0 2024-03-29T11:20:03.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.249+0000 7f92ea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:03.225 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.249+0000 7f92ea7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:03.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.581+0000 7f92fba8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:03.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.581+0000 7f92fba8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:03.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.581+0000 7f92fba8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:20:03.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.581+0000 7f92fba8f640 1 -- 172.21.15.170:0/1117264430 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92f40b74f0 con 0x7f92f41422b0 2024-03-29T11:20:03.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.582+0000 7f92ea7fc640 1 -- 172.21.15.170:0/1117264430 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f92ec0526b0 con 0x7f92f41422b0 2024-03-29T11:20:03.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.582+0000 7f92ea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:03.558 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.582+0000 7f92ea7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:03.559 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:20:03.559 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:03.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.586+0000 7f92fba8f640 10 librados: watch_flush enter 2024-03-29T11:20:03.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.586+0000 7f92fba8f640 10 librados: watch_flush exit 2024-03-29T11:20:03.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.586+0000 7f92fba8f640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-29T11:20:03.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.586+0000 7f92fba8f640 10 client.4571.objecter successfully canceled tick 2024-03-29T11:20:03.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.587+0000 7f92fba8f640 1 -- 172.21.15.170:0/1117264430 >> v1:172.21.15.170:6816/3298994444 conn(0x7f92d806dcd0 legacy=0x7f92d8070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:03.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.587+0000 7f92fba8f640 10 monclient: shutdown 2024-03-29T11:20:03.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.587+0000 7f92fba8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:03.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.587+0000 7f92fba8f640 1 -- 172.21.15.170:0/1117264430 >> v1:172.21.15.170:6790/0 conn(0x7f92f41422b0 legacy=0x7f92f40b5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:03.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.587+0000 7f92fba8f640 1 -- 172.21.15.170:0/1117264430 shutdown_connections 2024-03-29T11:20:03.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.587+0000 7f92fba8f640 1 -- 172.21.15.170:0/1117264430 >> 172.21.15.170:0/1117264430 conn(0x7f92f4066a00 msgr2=0x7f92f413e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:03.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.587+0000 7f92fba8f640 1 -- 172.21.15.170:0/1117264430 shutdown_connections 2024-03-29T11:20:03.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.587+0000 7f92fba8f640 1 -- 172.21.15.170:0/1117264430 wait complete. 2024-03-29T11:20:03.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:03.587+0000 7f92fba8f640 1 librados: shutdown 2024-03-29T11:20:03.582 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:10.389 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:10.539 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.561+0000 7f3c857ad640 10 monclient: get_monmap_and_config 2024-03-29T11:20:10.539 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.561+0000 7f3c857ad640 10 monclient: build_initial_monmap 2024-03-29T11:20:10.539 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.561+0000 7f3c857ad640 10 monclient: monmap: 2024-03-29T11:20:10.539 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:20:10.539 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:10.539 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:20:10.562896+0000 2024-03-29T11:20:10.539 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:20:10.562896+0000 2024-03-29T11:20:10.539 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:10.539 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:10.539 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:20:10.540 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:20:10.540 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:20:10.540 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:10.542 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.565+0000 7f3c857ad640 1 Processor -- start 2024-03-29T11:20:10.542 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.565+0000 7f3c857ad640 1 -- start start 2024-03-29T11:20:10.542 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.565+0000 7f3c857ad640 10 monclient: init 2024-03-29T11:20:10.543 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.565+0000 7f3c857ad640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:10.543 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.565+0000 7f3c857ad640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:10.543 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.566+0000 7f3c857ad640 10 monclient(hunting): picked mon.noname-a con 0x7f3c80071e60 addr v1:172.21.15.170:6789/0 2024-03-29T11:20:10.543 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.566+0000 7f3c857ad640 10 monclient(hunting): picked mon.noname-b con 0x7f3c800a9c70 addr v1:172.21.15.170:6790/0 2024-03-29T11:20:10.543 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.566+0000 7f3c857ad640 10 monclient(hunting): picked mon.noname-c con 0x7f3c8013f7e0 addr v1:172.21.15.184:6789/0 2024-03-29T11:20:10.543 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.566+0000 7f3c857ad640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c800731b0 con 0x7f3c80071e60 2024-03-29T11:20:10.543 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.566+0000 7f3c857ad640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c800aaeb0 con 0x7f3c8013f7e0 2024-03-29T11:20:10.544 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.566+0000 7f3c857ad640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c80072c90 con 0x7f3c800a9c70 2024-03-29T11:20:10.544 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.566+0000 7f3c857ad640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:10.544 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.566+0000 7f3c857ad640 10 monclient(hunting): authenticate will time out at 1391.878359s 2024-03-29T11:20:10.544 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.567+0000 7f3c7f7fe640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f3c800a9c70 0x7f3c801453b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:55622/0 (socket says 172.21.15.170:55622) 2024-03-29T11:20:10.544 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.567+0000 7f3c7f7fe640 1 -- 172.21.15.170:0/3255164794 learned_addr learned my addr 172.21.15.170:0/3255164794 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:20:10.544 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.567+0000 7f3c7effd640 1 -- 172.21.15.170:0/3255164794 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182088990 0 0) 0x7f3c80072c90 con 0x7f3c800a9c70 2024-03-29T11:20:10.545 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.567+0000 7f3c7effd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:10.545 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.567+0000 7f3c7effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:10.545 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.567+0000 7f3c7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:10.545 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.567+0000 7f3c7effd640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:20:10.545 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.567+0000 7f3c7effd640 1 -- 172.21.15.170:0/3255164794 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c6c003520 con 0x7f3c800a9c70 2024-03-29T11:20:10.545 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.567+0000 7f3c7effd640 1 -- 172.21.15.170:0/3255164794 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 492383000 0 0) 0x7f3c6c003520 con 0x7f3c800a9c70 2024-03-29T11:20:10.545 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.567+0000 7f3c7effd640 1 -- 172.21.15.170:0/3255164794 >> v1:172.21.15.184:6789/0 conn(0x7f3c8013f7e0 legacy=0x7f3c8014d710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:10.546 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.569+0000 7f3c7effd640 1 -- 172.21.15.170:0/3255164794 >> v1:172.21.15.170:6789/0 conn(0x7f3c80071e60 legacy=0x7f3c8013f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:10.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.570+0000 7f3c7effd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:10.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.570+0000 7f3c7effd640 1 monclient: found mon.noname-b 2024-03-29T11:20:10.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.570+0000 7f3c7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:10.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.570+0000 7f3c7effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:20:10.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.570+0000 7f3c7effd640 1 -- 172.21.15.170:0/3255164794 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c80145ac0 con 0x7f3c800a9c70 2024-03-29T11:20:10.550 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.570+0000 7f3c7effd640 10 monclient: _finish_auth 0 2024-03-29T11:20:10.550 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.570+0000 7f3c7effd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:10.550 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.570+0000 7f3c7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:10.550 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.570+0000 7f3c7effd640 1 -- 172.21.15.170:0/3255164794 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3c70001690 con 0x7f3c800a9c70 2024-03-29T11:20:10.550 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.570+0000 7f3c7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:10.550 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.571+0000 7f3c7effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:10.550 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.571+0000 7f3c7effd640 10 monclient: dump: 2024-03-29T11:20:10.550 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:10.550 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:10.550 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:10.551 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:10.551 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:10.551 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:10.551 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:10.551 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:10.551 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:10.551 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:10.551 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.571+0000 7f3c857ad640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:20:10.551 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.571+0000 7f3c857ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:10.551 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.571+0000 7f3c7effd640 1 -- 172.21.15.170:0/3255164794 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c700018f0 con 0x7f3c800a9c70 2024-03-29T11:20:10.551 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.571+0000 7f3c7effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:10.551 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.571+0000 7f3c7effd640 1 -- 172.21.15.170:0/3255164794 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3c70001d40 con 0x7f3c800a9c70 2024-03-29T11:20:10.551 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.571+0000 7f3c857ad640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:10.551 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.571+0000 7f3c857ad640 4 set_mon_vals no callback set 2024-03-29T11:20:10.551 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.571+0000 7f3c7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:10.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.571+0000 7f3c7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:10.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.571+0000 7f3c7effd640 10 monclient: dump: 2024-03-29T11:20:10.552 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:10.552 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:10.552 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:10.552 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:10.552 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:10.552 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:10.552 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:10.552 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:10.552 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:10.552 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:10.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.572+0000 7f3c857ad640 10 monclient: shutdown 2024-03-29T11:20:10.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.572+0000 7f3c857ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:10.553 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.572+0000 7f3c857ad640 1 -- 172.21.15.170:0/3255164794 >> v1:172.21.15.170:6790/0 conn(0x7f3c800a9c70 legacy=0x7f3c801453b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:10.553 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.572+0000 7f3c857ad640 1 -- 172.21.15.170:0/3255164794 shutdown_connections 2024-03-29T11:20:10.553 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.572+0000 7f3c857ad640 1 -- 172.21.15.170:0/3255164794 >> 172.21.15.170:0/3255164794 conn(0x7f3c800b3a30 msgr2=0x7f3c800b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:10.553 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.573+0000 7f3c857ad640 1 -- 172.21.15.170:0/3255164794 shutdown_connections 2024-03-29T11:20:10.553 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.573+0000 7f3c857ad640 1 -- 172.21.15.170:0/3255164794 wait complete. 2024-03-29T11:20:10.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.574+0000 7f3c857ad640 10 monclient: build_initial_monmap 2024-03-29T11:20:10.555 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.574+0000 7f3c857ad640 10 monclient: monmap: 2024-03-29T11:20:10.555 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:20:10.555 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:10.555 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:20:10.555 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:20:10.555 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:10.556 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:10.556 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:20:10.556 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:20:10.556 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:20:10.556 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:10.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.574+0000 7f3c857ad640 1 librados: starting msgr at 2024-03-29T11:20:10.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.574+0000 7f3c857ad640 1 librados: starting objecter 2024-03-29T11:20:10.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.575+0000 7f3c857ad640 1 Processor -- start 2024-03-29T11:20:10.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.575+0000 7f3c857ad640 1 -- start start 2024-03-29T11:20:10.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.575+0000 7f3c857ad640 1 librados: setting wanted keys 2024-03-29T11:20:10.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.575+0000 7f3c857ad640 1 librados: calling monclient init 2024-03-29T11:20:10.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.575+0000 7f3c857ad640 10 monclient: init 2024-03-29T11:20:10.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.576+0000 7f3c857ad640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:10.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.576+0000 7f3c857ad640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:10.556 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.576+0000 7f3c857ad640 10 monclient(hunting): picked mon.noname-a con 0x7f3c80071e60 addr v1:172.21.15.170:6789/0 2024-03-29T11:20:10.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.577+0000 7f3c857ad640 10 monclient(hunting): picked mon.noname-b con 0x7f3c800a9c70 addr v1:172.21.15.184:6789/0 2024-03-29T11:20:10.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.577+0000 7f3c857ad640 10 monclient(hunting): picked mon.noname-c con 0x7f3c8013f7e0 addr v1:172.21.15.170:6790/0 2024-03-29T11:20:10.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.577+0000 7f3c84fac640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f3c8013f7e0 0x7f3c801d7380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:33872/0 (socket says 172.21.15.170:33872) 2024-03-29T11:20:10.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.578+0000 7f3c84fac640 1 -- 172.21.15.170:0/1914185874 learned_addr learned my addr 172.21.15.170:0/1914185874 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:20:10.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.578+0000 7f3c857ad640 1 -- 172.21.15.170:0/1914185874 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8011b7f0 con 0x7f3c80071e60 2024-03-29T11:20:10.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.578+0000 7f3c857ad640 1 -- 172.21.15.170:0/1914185874 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c80144780 con 0x7f3c800a9c70 2024-03-29T11:20:10.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.578+0000 7f3c857ad640 1 -- 172.21.15.170:0/1914185874 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c80145290 con 0x7f3c8013f7e0 2024-03-29T11:20:10.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.578+0000 7f3c857ad640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:10.557 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.578+0000 7f3c857ad640 10 monclient(hunting): authenticate will time out at 1391.890462s 2024-03-29T11:20:10.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.580+0000 7f3c7cff9640 10 client.?.objecter ms_handle_connect 0x7f3c8013f7e0 2024-03-29T11:20:10.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.580+0000 7f3c7cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:10.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.580+0000 7f3c7cff9640 10 client.?.objecter ms_handle_connect 0x7f3c800a9c70 2024-03-29T11:20:10.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.581+0000 7f3c7cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:10.562 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.581+0000 7f3c7cff9640 1 -- 172.21.15.170:0/1914185874 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752630685 0 0) 0x7f3c80144780 con 0x7f3c800a9c70 2024-03-29T11:20:10.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.581+0000 7f3c7cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:10.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.581+0000 7f3c7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:10.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.581+0000 7f3c7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:10.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.581+0000 7f3c7cff9640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:20:10.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.581+0000 7f3c7cff9640 1 -- 172.21.15.170:0/1914185874 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c60004eb0 con 0x7f3c800a9c70 2024-03-29T11:20:10.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.581+0000 7f3c7cff9640 1 -- 172.21.15.170:0/1914185874 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823450480 0 0) 0x7f3c80145290 con 0x7f3c8013f7e0 2024-03-29T11:20:10.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.581+0000 7f3c7cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:10.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.581+0000 7f3c7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:10.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.581+0000 7f3c7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:10.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.581+0000 7f3c7cff9640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:20:10.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.581+0000 7f3c7cff9640 1 -- 172.21.15.170:0/1914185874 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c80144780 con 0x7f3c8013f7e0 2024-03-29T11:20:10.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.582+0000 7f3c7cff9640 1 -- 172.21.15.170:0/1914185874 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2742209282 0 0) 0x7f3c60004eb0 con 0x7f3c800a9c70 2024-03-29T11:20:10.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.582+0000 7f3c7cff9640 1 -- 172.21.15.170:0/1914185874 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c80145290 con 0x7f3c800a9c70 2024-03-29T11:20:10.563 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.582+0000 7f3c7cff9640 1 -- 172.21.15.170:0/1914185874 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3c700014b0 con 0x7f3c800a9c70 2024-03-29T11:20:10.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.582+0000 7f3c7cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:10.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.582+0000 7f3c7cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:10.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.582+0000 7f3c7cff9640 10 monclient(hunting): dump: 2024-03-29T11:20:10.564 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:10.564 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:10.564 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:10.564 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:10.564 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:10.564 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:10.564 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:10.564 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:10.564 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:10.564 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:10.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.582+0000 7f3c7cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:10.564 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.582+0000 7f3c7cff9640 1 -- 172.21.15.170:0/1914185874 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1332726704 0 0) 0x7f3c80145290 con 0x7f3c800a9c70 2024-03-29T11:20:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.582+0000 7f3c7cff9640 1 -- 172.21.15.170:0/1914185874 >> v1:172.21.15.170:6790/0 conn(0x7f3c8013f7e0 legacy=0x7f3c801d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.582+0000 7f3c7cff9640 1 -- 172.21.15.170:0/1914185874 >> v1:172.21.15.170:6789/0 conn(0x7f3c80071e60 legacy=0x7f3c80143eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.582+0000 7f3c7cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:20:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.582+0000 7f3c7cff9640 1 monclient: found mon.b 2024-03-29T11:20:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.583+0000 7f3c7cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.583+0000 7f3c7cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:20:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.583+0000 7f3c7cff9640 1 -- 172.21.15.170:0/1914185874 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c80139180 con 0x7f3c800a9c70 2024-03-29T11:20:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.583+0000 7f3c7cff9640 10 monclient: _finish_auth 0 2024-03-29T11:20:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.583+0000 7f3c7cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:20:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.583+0000 7f3c7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.583+0000 7f3c857ad640 5 monclient: authenticate success, global_id 4492 2024-03-29T11:20:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.583+0000 7f3c857ad640 10 monclient: _renew_subs 2024-03-29T11:20:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.583+0000 7f3c857ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:20:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.583+0000 7f3c857ad640 1 -- 172.21.15.170:0/1914185874 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c80139430 con 0x7f3c800a9c70 2024-03-29T11:20:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.583+0000 7f3c857ad640 10 client.4492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:10.565 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.583+0000 7f3c857ad640 10 monclient: _renew_subs 2024-03-29T11:20:10.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.583+0000 7f3c857ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:20:10.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.584+0000 7f3c857ad640 1 -- 172.21.15.170:0/1914185874 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c801e2960 con 0x7f3c800a9c70 2024-03-29T11:20:10.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.584+0000 7f3c857ad640 1 librados: init done 2024-03-29T11:20:10.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.584+0000 7f3c7cff9640 1 -- 172.21.15.170:0/1914185874 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c70001610 con 0x7f3c800a9c70 2024-03-29T11:20:10.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.584+0000 7f3c7cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:10.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.584+0000 7f3c7cff9640 1 -- 172.21.15.170:0/1914185874 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3c70004e10 con 0x7f3c800a9c70 2024-03-29T11:20:10.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.584+0000 7f3c7e7fc640 4 set_mon_vals no callback set 2024-03-29T11:20:10.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.584+0000 7f3c7cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:10.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.584+0000 7f3c7cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:10.566 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.584+0000 7f3c7cff9640 10 monclient: dump: 2024-03-29T11:20:10.566 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:10.566 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:10.566 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:10.566 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:10.567 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:10.567 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:10.567 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:10.567 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:10.567 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:10.567 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:10.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.587+0000 7f3c857ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:10.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.587+0000 7f3c857ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:10.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.587+0000 7f3c857ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:20:10.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.587+0000 7f3c857ad640 1 -- 172.21.15.170:0/1914185874 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c64005080 con 0x7f3c800a9c70 2024-03-29T11:20:10.568 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.587+0000 7f3c7cff9640 1 -- 172.21.15.170:0/1914185874 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f3c7001ade0 con 0x7f3c800a9c70 2024-03-29T11:20:10.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.588+0000 7f3c7cff9640 1 -- 172.21.15.170:0/1914185874 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f3c70084bb0 con 0x7f3c800a9c70 2024-03-29T11:20:10.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.588+0000 7f3c7cff9640 10 client.4492.objecter ms_dispatch 0x7f3c80000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:10.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.588+0000 7f3c7cff9640 3 client.4492.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:10.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.588+0000 7f3c7cff9640 3 client.4492.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:10.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.588+0000 7f3c7ffff640 10 monclient: get_auth_request con 0x7f3c6006de90 auth_method 0 2024-03-29T11:20:10.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.589+0000 7f3c7cff9640 20 client.4492.objecter dump_active .. 0 homeless 2024-03-29T11:20:10.569 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.589+0000 7f3c7cff9640 10 client.4492.objecter ms_handle_connect 0x7f3c6006de90 2024-03-29T11:20:10.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.596+0000 7f3c7cff9640 1 -- 172.21.15.170:0/1914185874 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c70050490 con 0x7f3c800a9c70 2024-03-29T11:20:10.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.596+0000 7f3c7cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:10.573 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.596+0000 7f3c7cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:10.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.922+0000 7f3c857ad640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:10.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.922+0000 7f3c857ad640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:10.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.922+0000 7f3c857ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:20:10.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.922+0000 7f3c857ad640 1 -- 172.21.15.170:0/1914185874 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c64005330 con 0x7f3c800a9c70 2024-03-29T11:20:10.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.923+0000 7f3c7cff9640 1 -- 172.21.15.170:0/1914185874 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f3c700523c0 con 0x7f3c800a9c70 2024-03-29T11:20:10.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.923+0000 7f3c7cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:10.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.923+0000 7f3c7cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:10.901 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:20:10.901 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:10.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.930+0000 7f3c857ad640 10 librados: watch_flush enter 2024-03-29T11:20:10.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.931+0000 7f3c857ad640 10 librados: watch_flush exit 2024-03-29T11:20:10.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.931+0000 7f3c857ad640 20 client.4492.objecter shutdown clearing up homeless session... 2024-03-29T11:20:10.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.931+0000 7f3c857ad640 10 client.4492.objecter successfully canceled tick 2024-03-29T11:20:10.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.931+0000 7f3c857ad640 1 -- 172.21.15.170:0/1914185874 >> v1:172.21.15.170:6816/3298994444 conn(0x7f3c6006de90 legacy=0x7f3c60070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:10.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.931+0000 7f3c857ad640 10 monclient: shutdown 2024-03-29T11:20:10.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.931+0000 7f3c857ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:10.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.931+0000 7f3c857ad640 1 -- 172.21.15.170:0/1914185874 >> v1:172.21.15.184:6789/0 conn(0x7f3c800a9c70 legacy=0x7f3c801d6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:10.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.931+0000 7f3c857ad640 1 -- 172.21.15.170:0/1914185874 shutdown_connections 2024-03-29T11:20:10.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.931+0000 7f3c857ad640 1 -- 172.21.15.170:0/1914185874 >> 172.21.15.170:0/1914185874 conn(0x7f3c800b3a30 msgr2=0x7f3c8013cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:10.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.931+0000 7f3c857ad640 1 -- 172.21.15.170:0/1914185874 shutdown_connections 2024-03-29T11:20:10.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.931+0000 7f3c857ad640 1 -- 172.21.15.170:0/1914185874 wait complete. 2024-03-29T11:20:10.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:10.932+0000 7f3c857ad640 1 librados: shutdown 2024-03-29T11:20:10.927 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:17.733 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:17.886 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.908+0000 7efc706d9640 10 monclient: get_monmap_and_config 2024-03-29T11:20:17.886 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.908+0000 7efc706d9640 10 monclient: build_initial_monmap 2024-03-29T11:20:17.886 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.908+0000 7efc706d9640 10 monclient: monmap: 2024-03-29T11:20:17.886 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:20:17.886 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:17.886 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:20:17.909924+0000 2024-03-29T11:20:17.886 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:20:17.909924+0000 2024-03-29T11:20:17.886 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:17.886 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:17.887 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:20:17.887 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:20:17.887 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:20:17.887 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:17.888 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.909+0000 7efc706d9640 1 Processor -- start 2024-03-29T11:20:17.888 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.909+0000 7efc706d9640 1 -- start start 2024-03-29T11:20:17.888 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.909+0000 7efc706d9640 10 monclient: init 2024-03-29T11:20:17.888 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.910+0000 7efc706d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:17.888 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.910+0000 7efc706d9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:17.892 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.914+0000 7efc706d9640 10 monclient(hunting): picked mon.noname-b con 0x7efc680ae9b0 addr v1:172.21.15.170:6790/0 2024-03-29T11:20:17.892 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.914+0000 7efc706d9640 10 monclient(hunting): picked mon.noname-a con 0x7efc68143f50 addr v1:172.21.15.170:6789/0 2024-03-29T11:20:17.892 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.914+0000 7efc706d9640 10 monclient(hunting): picked mon.noname-c con 0x7efc680b4890 addr v1:172.21.15.184:6789/0 2024-03-29T11:20:17.892 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.914+0000 7efc706d9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc680aed90 con 0x7efc68143f50 2024-03-29T11:20:17.892 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.914+0000 7efc706d9640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc680af020 con 0x7efc680b4890 2024-03-29T11:20:17.892 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.914+0000 7efc706d9640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc680af300 con 0x7efc680ae9b0 2024-03-29T11:20:17.892 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.914+0000 7efc706d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:17.892 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.914+0000 7efc706d9640 10 monclient(hunting): authenticate will time out at 1399.226304s 2024-03-29T11:20:17.894 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.915+0000 7efc6e475640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7efc680ae9b0 0x7efc68143840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:55993/0 (socket says 172.21.15.170:55993) 2024-03-29T11:20:17.895 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.915+0000 7efc6e475640 1 -- 172.21.15.170:0/1912180411 learned_addr learned my addr 172.21.15.170:0/1912180411 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:20:17.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.920+0000 7efc6d473640 1 -- 172.21.15.170:0/1912180411 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916336740 0 0) 0x7efc680af300 con 0x7efc680ae9b0 2024-03-29T11:20:17.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.920+0000 7efc6d473640 10 monclient(hunting): _negotiate 2024-03-29T11:20:17.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.920+0000 7efc6d473640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:17.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.920+0000 7efc6d473640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:17.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.920+0000 7efc6d473640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:20:17.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.920+0000 7efc6d473640 1 -- 172.21.15.170:0/1912180411 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc54003520 con 0x7efc680ae9b0 2024-03-29T11:20:17.897 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.920+0000 7efc6d473640 1 -- 172.21.15.170:0/1912180411 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857359832 0 0) 0x7efc680aed90 con 0x7efc68143f50 2024-03-29T11:20:17.898 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.920+0000 7efc6d473640 10 monclient(hunting): _negotiate 2024-03-29T11:20:17.898 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.920+0000 7efc6d473640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:17.898 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.920+0000 7efc6d473640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:17.898 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.920+0000 7efc6d473640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:20:17.898 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.920+0000 7efc6d473640 1 -- 172.21.15.170:0/1912180411 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc680af300 con 0x7efc68143f50 2024-03-29T11:20:17.898 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.920+0000 7efc6d473640 1 -- 172.21.15.170:0/1912180411 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143658471 0 0) 0x7efc680af020 con 0x7efc680b4890 2024-03-29T11:20:17.898 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.920+0000 7efc6d473640 10 monclient(hunting): _negotiate 2024-03-29T11:20:17.898 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.920+0000 7efc6d473640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:17.898 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.920+0000 7efc6d473640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:17.898 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.920+0000 7efc6d473640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:20:17.898 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.920+0000 7efc6d473640 1 -- 172.21.15.170:0/1912180411 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc680aed90 con 0x7efc680b4890 2024-03-29T11:20:17.899 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 1 -- 172.21.15.170:0/1912180411 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 191924358 0 0) 0x7efc54003520 con 0x7efc680ae9b0 2024-03-29T11:20:17.899 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 1 -- 172.21.15.170:0/1912180411 >> v1:172.21.15.184:6789/0 conn(0x7efc680b4890 legacy=0x7efc68151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:17.899 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 1 -- 172.21.15.170:0/1912180411 >> v1:172.21.15.170:6789/0 conn(0x7efc68143f50 legacy=0x7efc680b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:17.899 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 10 monclient: _finish_hunting 0 2024-03-29T11:20:17.899 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 1 monclient: found mon.noname-b 2024-03-29T11:20:17.899 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:17.899 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:20:17.899 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 1 -- 172.21.15.170:0/1912180411 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc680b0710 con 0x7efc680ae9b0 2024-03-29T11:20:17.899 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 10 monclient: _finish_auth 0 2024-03-29T11:20:17.899 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 10 monclient: _check_auth_tickets 2024-03-29T11:20:17.899 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:17.899 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 1 -- 172.21.15.170:0/1912180411 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7efc64001690 con 0x7efc680ae9b0 2024-03-29T11:20:17.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:17.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:17.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 10 monclient: dump: 2024-03-29T11:20:17.900 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:17.900 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:17.900 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:17.900 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:17.900 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:17.900 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:17.900 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:17.900 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:17.900 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:17.900 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:17.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 1 -- 172.21.15.170:0/1912180411 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc64001870 con 0x7efc680ae9b0 2024-03-29T11:20:17.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:17.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 1 -- 172.21.15.170:0/1912180411 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7efc64001c00 con 0x7efc680ae9b0 2024-03-29T11:20:17.900 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:17.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:17.901 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.921+0000 7efc6d473640 10 monclient: dump: 2024-03-29T11:20:17.901 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:17.901 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:17.901 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:17.901 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:17.901 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:17.901 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:17.901 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:17.901 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:17.901 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:17.901 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:17.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.923+0000 7efc706d9640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:20:17.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.923+0000 7efc706d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:17.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.923+0000 7efc706d9640 4 set_mon_vals no callback set 2024-03-29T11:20:17.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.924+0000 7efc706d9640 10 monclient: shutdown 2024-03-29T11:20:17.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.924+0000 7efc706d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:17.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.924+0000 7efc706d9640 1 -- 172.21.15.170:0/1912180411 >> v1:172.21.15.170:6790/0 conn(0x7efc680ae9b0 legacy=0x7efc68143840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:17.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.924+0000 7efc706d9640 1 -- 172.21.15.170:0/1912180411 shutdown_connections 2024-03-29T11:20:17.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.924+0000 7efc706d9640 1 -- 172.21.15.170:0/1912180411 >> 172.21.15.170:0/1912180411 conn(0x7efc68066a00 msgr2=0x7efc68066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:17.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.924+0000 7efc706d9640 1 -- 172.21.15.170:0/1912180411 shutdown_connections 2024-03-29T11:20:17.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.924+0000 7efc706d9640 1 -- 172.21.15.170:0/1912180411 wait complete. 2024-03-29T11:20:17.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.925+0000 7efc706d9640 10 monclient: build_initial_monmap 2024-03-29T11:20:17.903 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.925+0000 7efc706d9640 10 monclient: monmap: 2024-03-29T11:20:17.904 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:20:17.904 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:17.904 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:20:17.904 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:20:17.904 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:17.904 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:17.904 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:20:17.904 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:20:17.904 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:20:17.904 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:17.904 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.925+0000 7efc706d9640 1 librados: starting msgr at 2024-03-29T11:20:17.904 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.925+0000 7efc706d9640 1 librados: starting objecter 2024-03-29T11:20:17.904 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.926+0000 7efc706d9640 1 Processor -- start 2024-03-29T11:20:17.904 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.926+0000 7efc706d9640 1 -- start start 2024-03-29T11:20:17.904 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.926+0000 7efc706d9640 1 librados: setting wanted keys 2024-03-29T11:20:17.905 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.926+0000 7efc706d9640 1 librados: calling monclient init 2024-03-29T11:20:17.905 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.926+0000 7efc706d9640 10 monclient: init 2024-03-29T11:20:17.906 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.927+0000 7efc706d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:17.906 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.927+0000 7efc706d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:17.906 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.927+0000 7efc706d9640 10 monclient(hunting): picked mon.noname-b con 0x7efc680ae9b0 addr v1:172.21.15.184:6789/0 2024-03-29T11:20:17.906 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.927+0000 7efc706d9640 10 monclient(hunting): picked mon.noname-a con 0x7efc680b4890 addr v1:172.21.15.170:6789/0 2024-03-29T11:20:17.906 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.927+0000 7efc706d9640 10 monclient(hunting): picked mon.noname-c con 0x7efc68143f50 addr v1:172.21.15.170:6790/0 2024-03-29T11:20:17.906 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.927+0000 7efc706d9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc681db410 con 0x7efc680b4890 2024-03-29T11:20:17.906 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.927+0000 7efc706d9640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc6806ffe0 con 0x7efc680ae9b0 2024-03-29T11:20:17.906 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.927+0000 7efc706d9640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc680702c0 con 0x7efc68143f50 2024-03-29T11:20:17.906 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.927+0000 7efc706d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:17.906 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.927+0000 7efc706d9640 10 monclient(hunting): authenticate will time out at 1399.239551s 2024-03-29T11:20:17.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.930+0000 7efc6e475640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7efc680ae9b0 0x7efc681db8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:42787/0 (socket says 172.21.15.170:42787) 2024-03-29T11:20:17.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.930+0000 7efc6e475640 1 -- 172.21.15.170:0/2656496373 learned_addr learned my addr 172.21.15.170:0/2656496373 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:20:17.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.930+0000 7efc5effd640 10 client.?.objecter ms_handle_connect 0x7efc680ae9b0 2024-03-29T11:20:17.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.930+0000 7efc5effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:17.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.930+0000 7efc5effd640 1 -- 172.21.15.170:0/2656496373 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714778956 0 0) 0x7efc6806ffe0 con 0x7efc680ae9b0 2024-03-29T11:20:17.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.930+0000 7efc5effd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:17.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.930+0000 7efc5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:17.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.930+0000 7efc5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:17.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.930+0000 7efc5effd640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:20:17.908 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.930+0000 7efc5effd640 1 -- 172.21.15.170:0/2656496373 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc4c004eb0 con 0x7efc680ae9b0 2024-03-29T11:20:17.909 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.932+0000 7efc5effd640 10 client.?.objecter ms_handle_connect 0x7efc680b4890 2024-03-29T11:20:17.909 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.932+0000 7efc5effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:17.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.932+0000 7efc5effd640 1 -- 172.21.15.170:0/2656496373 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467253002 0 0) 0x7efc681db410 con 0x7efc680b4890 2024-03-29T11:20:17.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.932+0000 7efc5effd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:17.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.932+0000 7efc5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:17.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.932+0000 7efc5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:17.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.932+0000 7efc5effd640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:20:17.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.932+0000 7efc5effd640 1 -- 172.21.15.170:0/2656496373 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc6806ffe0 con 0x7efc680b4890 2024-03-29T11:20:17.910 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.932+0000 7efc5effd640 1 -- 172.21.15.170:0/2656496373 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 650047509 0 0) 0x7efc6806ffe0 con 0x7efc680b4890 2024-03-29T11:20:17.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.932+0000 7efc5effd640 1 -- 172.21.15.170:0/2656496373 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc681db410 con 0x7efc680b4890 2024-03-29T11:20:17.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.932+0000 7efc5effd640 1 -- 172.21.15.170:0/2656496373 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7efc60001b40 con 0x7efc680b4890 2024-03-29T11:20:17.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.932+0000 7efc5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:17.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.932+0000 7efc5effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:17.911 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.932+0000 7efc5effd640 10 monclient(hunting): dump: 2024-03-29T11:20:17.911 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:17.911 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:17.911 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:17.911 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:17.911 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:17.911 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:17.911 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:17.911 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:17.911 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:17.911 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:17.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.932+0000 7efc5effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:17.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc5effd640 1 -- 172.21.15.170:0/2656496373 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3617491097 0 0) 0x7efc681db410 con 0x7efc680b4890 2024-03-29T11:20:17.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc5effd640 1 -- 172.21.15.170:0/2656496373 >> v1:172.21.15.170:6790/0 conn(0x7efc68143f50 legacy=0x7efc681e5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:17.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc5effd640 1 -- 172.21.15.170:0/2656496373 >> v1:172.21.15.184:6789/0 conn(0x7efc680ae9b0 legacy=0x7efc681db8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:17.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc5effd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:17.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc5effd640 1 monclient: found mon.a 2024-03-29T11:20:17.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:17.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc5effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:20:17.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc5effd640 1 -- 172.21.15.170:0/2656496373 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc681dbfb0 con 0x7efc680b4890 2024-03-29T11:20:17.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc5effd640 10 monclient: _finish_auth 0 2024-03-29T11:20:17.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc5effd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:17.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:17.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc706d9640 5 monclient: authenticate success, global_id 4530 2024-03-29T11:20:17.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc706d9640 10 monclient: _renew_subs 2024-03-29T11:20:17.912 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc706d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:20:17.913 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc706d9640 1 -- 172.21.15.170:0/2656496373 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc68070a70 con 0x7efc680b4890 2024-03-29T11:20:17.913 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc706d9640 10 client.4530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:17.913 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc706d9640 10 monclient: _renew_subs 2024-03-29T11:20:17.913 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc706d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:20:17.913 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc706d9640 1 -- 172.21.15.170:0/2656496373 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc6806f1a0 con 0x7efc680b4890 2024-03-29T11:20:17.913 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.933+0000 7efc706d9640 1 librados: init done 2024-03-29T11:20:17.914 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.934+0000 7efc5effd640 1 -- 172.21.15.170:0/2656496373 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc60001ff0 con 0x7efc680b4890 2024-03-29T11:20:17.914 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.934+0000 7efc5effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:17.914 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.934+0000 7efc5effd640 1 -- 172.21.15.170:0/2656496373 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7efc60001b40 con 0x7efc680b4890 2024-03-29T11:20:17.914 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.934+0000 7efc5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:17.914 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.934+0000 7efc5effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:17.914 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.934+0000 7efc5effd640 10 monclient: dump: 2024-03-29T11:20:17.914 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:17.915 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:17.915 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:17.915 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:17.915 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:17.915 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:17.915 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:17.915 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:17.915 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:17.915 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:17.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.934+0000 7efc6cc72640 4 set_mon_vals no callback set 2024-03-29T11:20:17.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.934+0000 7efc706d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:17.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.935+0000 7efc706d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:17.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.935+0000 7efc706d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:20:17.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.935+0000 7efc706d9640 1 -- 172.21.15.170:0/2656496373 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc50005080 con 0x7efc680b4890 2024-03-29T11:20:17.915 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.936+0000 7efc5effd640 1 -- 172.21.15.170:0/2656496373 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7efc600197e0 con 0x7efc680b4890 2024-03-29T11:20:17.916 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.936+0000 7efc5effd640 1 -- 172.21.15.170:0/2656496373 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7efc600535d0 con 0x7efc680b4890 2024-03-29T11:20:17.916 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.936+0000 7efc5effd640 10 client.4530.objecter ms_dispatch 0x7efc68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:17.916 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.936+0000 7efc5effd640 3 client.4530.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:17.916 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.936+0000 7efc5effd640 3 client.4530.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:17.916 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.936+0000 7efc5effd640 20 client.4530.objecter dump_active .. 0 homeless 2024-03-29T11:20:17.916 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.939+0000 7efc6e475640 10 monclient: get_auth_request con 0x7efc4c06dc70 auth_method 0 2024-03-29T11:20:17.917 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.940+0000 7efc5effd640 10 client.4530.objecter ms_handle_connect 0x7efc4c06dc70 2024-03-29T11:20:17.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.944+0000 7efc5effd640 1 -- 172.21.15.170:0/2656496373 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc6004f870 con 0x7efc680b4890 2024-03-29T11:20:17.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.944+0000 7efc5effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:17.922 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:17.944+0000 7efc5effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:18.245 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.267+0000 7efc706d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:18.245 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.267+0000 7efc706d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:18.245 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.267+0000 7efc706d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:20:18.245 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.267+0000 7efc706d9640 1 -- 172.21.15.170:0/2656496373 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc50005330 con 0x7efc680b4890 2024-03-29T11:20:18.247 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.270+0000 7efc5effd640 1 -- 172.21.15.170:0/2656496373 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7efc6004f870 con 0x7efc680b4890 2024-03-29T11:20:18.247 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.270+0000 7efc5effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:18.247 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.270+0000 7efc5effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:18.247 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:20:18.247 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:18.250 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.273+0000 7efc706d9640 10 librados: watch_flush enter 2024-03-29T11:20:18.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.274+0000 7efc706d9640 10 librados: watch_flush exit 2024-03-29T11:20:18.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.274+0000 7efc706d9640 20 client.4530.objecter shutdown clearing up homeless session... 2024-03-29T11:20:18.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.274+0000 7efc706d9640 10 client.4530.objecter successfully canceled tick 2024-03-29T11:20:18.252 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.274+0000 7efc706d9640 1 -- 172.21.15.170:0/2656496373 >> v1:172.21.15.170:6816/3298994444 conn(0x7efc4c06dc70 legacy=0x7efc4c070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:18.252 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.275+0000 7efc706d9640 10 monclient: shutdown 2024-03-29T11:20:18.252 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.275+0000 7efc706d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:18.252 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.275+0000 7efc706d9640 1 -- 172.21.15.170:0/2656496373 >> v1:172.21.15.170:6789/0 conn(0x7efc680b4890 legacy=0x7efc6806e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:18.252 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.275+0000 7efc706d9640 1 -- 172.21.15.170:0/2656496373 shutdown_connections 2024-03-29T11:20:18.252 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.275+0000 7efc706d9640 1 -- 172.21.15.170:0/2656496373 >> 172.21.15.170:0/2656496373 conn(0x7efc68066a00 msgr2=0x7efc680b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:18.255 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.278+0000 7efc706d9640 1 -- 172.21.15.170:0/2656496373 shutdown_connections 2024-03-29T11:20:18.255 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.278+0000 7efc706d9640 1 -- 172.21.15.170:0/2656496373 wait complete. 2024-03-29T11:20:18.255 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:18.278+0000 7efc706d9640 1 librados: shutdown 2024-03-29T11:20:18.273 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:25.081 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:25.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.274+0000 7f99400a5640 10 monclient: get_monmap_and_config 2024-03-29T11:20:25.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.274+0000 7f99400a5640 10 monclient: build_initial_monmap 2024-03-29T11:20:25.251 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.274+0000 7f99400a5640 10 monclient: monmap: 2024-03-29T11:20:25.251 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:20:25.252 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:25.252 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:20:25.275721+0000 2024-03-29T11:20:25.252 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:20:25.275721+0000 2024-03-29T11:20:25.252 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:25.252 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:25.252 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:20:25.252 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:20:25.252 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:20:25.252 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:25.253 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.275+0000 7f99400a5640 1 Processor -- start 2024-03-29T11:20:25.253 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.275+0000 7f99400a5640 1 -- start start 2024-03-29T11:20:25.253 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.275+0000 7f99400a5640 10 monclient: init 2024-03-29T11:20:25.253 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.276+0000 7f99400a5640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:25.254 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.276+0000 7f99400a5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:25.254 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.276+0000 7f99400a5640 10 monclient(hunting): picked mon.noname-c con 0x7f99381457f0 addr v1:172.21.15.184:6789/0 2024-03-29T11:20:25.258 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.280+0000 7f99400a5640 10 monclient(hunting): picked mon.noname-a con 0x7f993813f650 addr v1:172.21.15.170:6789/0 2024-03-29T11:20:25.258 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.280+0000 7f993de41640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f99381457f0 0x7f993813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:59401/0 (socket says 172.21.15.170:59401) 2024-03-29T11:20:25.258 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.280+0000 7f993de41640 1 -- 172.21.15.170:0/2081505199 learned_addr learned my addr 172.21.15.170:0/2081505199 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:20:25.258 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.280+0000 7f99400a5640 10 monclient(hunting): picked mon.noname-b con 0x7f99380a9d70 addr v1:172.21.15.170:6790/0 2024-03-29T11:20:25.258 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.280+0000 7f99400a5640 1 -- 172.21.15.170:0/2081505199 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99380aaef0 con 0x7f993813f650 2024-03-29T11:20:25.258 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.280+0000 7f99400a5640 1 -- 172.21.15.170:0/2081505199 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9938145bd0 con 0x7f99381457f0 2024-03-29T11:20:25.258 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.280+0000 7f99400a5640 1 -- 172.21.15.170:0/2081505199 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9938145eb0 con 0x7f99380a9d70 2024-03-29T11:20:25.258 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.280+0000 7f99400a5640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:25.258 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.280+0000 7f99400a5640 10 monclient(hunting): authenticate will time out at 1406.592771s 2024-03-29T11:20:25.258 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.280+0000 7f993ce3f640 1 -- 172.21.15.170:0/2081505199 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205511341 0 0) 0x7f9938145bd0 con 0x7f99381457f0 2024-03-29T11:20:25.258 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.281+0000 7f993ce3f640 10 monclient(hunting): _negotiate 2024-03-29T11:20:25.259 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.281+0000 7f993ce3f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:25.259 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.281+0000 7f993ce3f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:25.259 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.281+0000 7f993ce3f640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:20:25.259 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.281+0000 7f993ce3f640 1 -- 172.21.15.170:0/2081505199 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9924003520 con 0x7f99381457f0 2024-03-29T11:20:25.262 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.285+0000 7f993ce3f640 1 -- 172.21.15.170:0/2081505199 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 509857695 0 0) 0x7f9924003520 con 0x7f99381457f0 2024-03-29T11:20:25.262 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.285+0000 7f993ce3f640 1 -- 172.21.15.170:0/2081505199 >> v1:172.21.15.170:6790/0 conn(0x7f99380a9d70 legacy=0x7f993814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:25.262 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.285+0000 7f993ce3f640 1 -- 172.21.15.170:0/2081505199 >> v1:172.21.15.170:6789/0 conn(0x7f993813f650 legacy=0x7f99380b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:25.262 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.285+0000 7f993ce3f640 10 monclient: _finish_hunting 0 2024-03-29T11:20:25.263 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.285+0000 7f993ce3f640 1 monclient: found mon.noname-c 2024-03-29T11:20:25.263 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.285+0000 7f993ce3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:25.263 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.285+0000 7f993ce3f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:20:25.263 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.285+0000 7f993ce3f640 1 -- 172.21.15.170:0/2081505199 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9938146340 con 0x7f99381457f0 2024-03-29T11:20:25.263 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.285+0000 7f993ce3f640 10 monclient: _finish_auth 0 2024-03-29T11:20:25.263 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.285+0000 7f993ce3f640 10 monclient: _check_auth_tickets 2024-03-29T11:20:25.263 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.285+0000 7f993ce3f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:25.263 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.285+0000 7f993ce3f640 1 -- 172.21.15.170:0/2081505199 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9928001690 con 0x7f99381457f0 2024-03-29T11:20:25.263 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.285+0000 7f993ce3f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:25.263 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.285+0000 7f993ce3f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:25.263 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.285+0000 7f993ce3f640 10 monclient: dump: 2024-03-29T11:20:25.263 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:25.263 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:25.263 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:25.263 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:25.264 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:25.264 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:25.264 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:25.264 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:25.264 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:25.264 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:25.264 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.285+0000 7f99400a5640 5 monclient: authenticate success, global_id 4501 2024-03-29T11:20:25.264 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.285+0000 7f99400a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:25.269 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.287+0000 7f993ce3f640 1 -- 172.21.15.170:0/2081505199 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9928001920 con 0x7f99381457f0 2024-03-29T11:20:25.269 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.287+0000 7f993ce3f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:25.269 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.287+0000 7f993ce3f640 1 -- 172.21.15.170:0/2081505199 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9928001cb0 con 0x7f99381457f0 2024-03-29T11:20:25.269 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.287+0000 7f993ce3f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:25.269 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.287+0000 7f993ce3f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:25.269 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.287+0000 7f993ce3f640 10 monclient: dump: 2024-03-29T11:20:25.269 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:25.269 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:25.270 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:25.270 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:25.270 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:25.270 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:25.270 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:25.270 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:25.270 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:25.270 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:25.270 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.291+0000 7f99400a5640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:25.270 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.292+0000 7f99400a5640 4 set_mon_vals no callback set 2024-03-29T11:20:25.271 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.293+0000 7f99400a5640 10 monclient: shutdown 2024-03-29T11:20:25.271 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.293+0000 7f99400a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:25.271 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.293+0000 7f99400a5640 1 -- 172.21.15.170:0/2081505199 >> v1:172.21.15.184:6789/0 conn(0x7f99381457f0 legacy=0x7f993813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:25.271 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.293+0000 7f99400a5640 1 -- 172.21.15.170:0/2081505199 shutdown_connections 2024-03-29T11:20:25.271 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.293+0000 7f99400a5640 1 -- 172.21.15.170:0/2081505199 >> 172.21.15.170:0/2081505199 conn(0x7f9938066a00 msgr2=0x7f9938066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:25.272 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.294+0000 7f99400a5640 1 -- 172.21.15.170:0/2081505199 shutdown_connections 2024-03-29T11:20:25.272 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.295+0000 7f99400a5640 1 -- 172.21.15.170:0/2081505199 wait complete. 2024-03-29T11:20:25.273 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.296+0000 7f99400a5640 10 monclient: build_initial_monmap 2024-03-29T11:20:25.273 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.296+0000 7f99400a5640 10 monclient: monmap: 2024-03-29T11:20:25.273 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:20:25.273 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:25.274 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:20:25.274 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:20:25.274 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:25.274 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:25.274 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:20:25.274 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:20:25.274 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:20:25.274 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:25.276 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.297+0000 7f99400a5640 1 librados: starting msgr at 2024-03-29T11:20:25.276 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.297+0000 7f99400a5640 1 librados: starting objecter 2024-03-29T11:20:25.276 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.297+0000 7f99400a5640 1 Processor -- start 2024-03-29T11:20:25.276 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.297+0000 7f99400a5640 1 -- start start 2024-03-29T11:20:25.276 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.297+0000 7f99400a5640 1 librados: setting wanted keys 2024-03-29T11:20:25.276 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.297+0000 7f99400a5640 1 librados: calling monclient init 2024-03-29T11:20:25.276 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.297+0000 7f99400a5640 10 monclient: init 2024-03-29T11:20:25.276 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.298+0000 7f99400a5640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:25.276 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.298+0000 7f99400a5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:25.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.300+0000 7f99400a5640 10 monclient(hunting): picked mon.noname-b con 0x7f99380a9d70 addr v1:172.21.15.184:6789/0 2024-03-29T11:20:25.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.300+0000 7f99400a5640 10 monclient(hunting): picked mon.noname-c con 0x7f993813f650 addr v1:172.21.15.170:6790/0 2024-03-29T11:20:25.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.300+0000 7f99400a5640 10 monclient(hunting): picked mon.noname-a con 0x7f99381457f0 addr v1:172.21.15.170:6789/0 2024-03-29T11:20:25.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.300+0000 7f99400a5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99381da720 con 0x7f99381457f0 2024-03-29T11:20:25.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.300+0000 7f99400a5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9938135a00 con 0x7f99380a9d70 2024-03-29T11:20:25.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.300+0000 7f99400a5640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993813aa20 con 0x7f993813f650 2024-03-29T11:20:25.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.300+0000 7f99400a5640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:25.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.300+0000 7f99400a5640 10 monclient(hunting): authenticate will time out at 1406.612997s 2024-03-29T11:20:25.283 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.306+0000 7f993de41640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f99380a9d70 0x7f99381dabb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:59507/0 (socket says 172.21.15.170:59507) 2024-03-29T11:20:25.283 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.306+0000 7f993e642640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f99381457f0 0x7f99381352f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:44285/0 (socket says 172.21.15.170:44285) 2024-03-29T11:20:25.283 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.306+0000 7f993e642640 1 -- 172.21.15.170:0/1157040525 learned_addr learned my addr 172.21.15.170:0/1157040525 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:20:25.284 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.306+0000 7f992e7fc640 10 client.?.objecter ms_handle_connect 0x7f99381457f0 2024-03-29T11:20:25.284 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.306+0000 7f992e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:25.284 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.306+0000 7f992e7fc640 10 client.?.objecter ms_handle_connect 0x7f99380a9d70 2024-03-29T11:20:25.284 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.307+0000 7f992e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:25.284 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.307+0000 7f992e7fc640 10 client.?.objecter ms_handle_connect 0x7f993813f650 2024-03-29T11:20:25.284 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.307+0000 7f992e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:25.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 1 -- 172.21.15.170:0/1157040525 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702286964 0 0) 0x7f993813aa20 con 0x7f993813f650 2024-03-29T11:20:25.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:25.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:25.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:25.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:20:25.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 1 -- 172.21.15.170:0/1157040525 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f991c004eb0 con 0x7f993813f650 2024-03-29T11:20:25.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 1 -- 172.21.15.170:0/1157040525 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266505477 0 0) 0x7f9938135a00 con 0x7f99380a9d70 2024-03-29T11:20:25.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:25.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:25.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:25.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:20:25.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 1 -- 172.21.15.170:0/1157040525 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f993813aa20 con 0x7f99380a9d70 2024-03-29T11:20:25.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 1 -- 172.21.15.170:0/1157040525 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540854504 0 0) 0x7f99381da720 con 0x7f99381457f0 2024-03-29T11:20:25.291 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:25.291 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:25.291 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:25.291 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:20:25.291 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 1 -- 172.21.15.170:0/1157040525 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9938135a00 con 0x7f99381457f0 2024-03-29T11:20:25.291 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 1 -- 172.21.15.170:0/1157040525 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 421063047 0 0) 0x7f993813aa20 con 0x7f99380a9d70 2024-03-29T11:20:25.291 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.310+0000 7f992e7fc640 1 -- 172.21.15.170:0/1157040525 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99381da720 con 0x7f99380a9d70 2024-03-29T11:20:25.291 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f992e7fc640 1 -- 172.21.15.170:0/1157040525 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9928001550 con 0x7f99380a9d70 2024-03-29T11:20:25.291 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f992e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:25.291 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f992e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:25.291 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f992e7fc640 10 monclient(hunting): dump: 2024-03-29T11:20:25.291 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:25.291 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:25.292 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:25.292 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:25.292 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:25.292 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:25.292 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:25.292 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:25.292 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:25.292 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:25.292 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f992e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:25.292 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f992e7fc640 1 -- 172.21.15.170:0/1157040525 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3718007056 0 0) 0x7f99381da720 con 0x7f99380a9d70 2024-03-29T11:20:25.292 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f992e7fc640 1 -- 172.21.15.170:0/1157040525 >> v1:172.21.15.170:6790/0 conn(0x7f993813f650 legacy=0x7f9938134be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:25.292 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f992e7fc640 1 -- 172.21.15.170:0/1157040525 >> v1:172.21.15.170:6789/0 conn(0x7f99381457f0 legacy=0x7f99381352f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:25.292 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f992e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:25.292 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f992e7fc640 1 monclient: found mon.b 2024-03-29T11:20:25.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f992e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:25.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f992e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:20:25.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f992e7fc640 1 -- 172.21.15.170:0/1157040525 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993813ad00 con 0x7f99380a9d70 2024-03-29T11:20:25.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f992e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:25.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f992e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:25.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f992e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:25.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f99400a5640 5 monclient: authenticate success, global_id 4504 2024-03-29T11:20:25.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f99400a5640 10 monclient: _renew_subs 2024-03-29T11:20:25.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f99400a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:20:25.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.311+0000 7f99400a5640 1 -- 172.21.15.170:0/1157040525 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99381402e0 con 0x7f99380a9d70 2024-03-29T11:20:25.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.312+0000 7f99400a5640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:25.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.312+0000 7f99400a5640 10 monclient: _renew_subs 2024-03-29T11:20:25.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.312+0000 7f99400a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:20:25.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.312+0000 7f99400a5640 1 -- 172.21.15.170:0/1157040525 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9938142b90 con 0x7f99380a9d70 2024-03-29T11:20:25.293 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.312+0000 7f99400a5640 1 librados: init done 2024-03-29T11:20:25.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.312+0000 7f992e7fc640 1 -- 172.21.15.170:0/1157040525 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99280016b0 con 0x7f99380a9d70 2024-03-29T11:20:25.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.312+0000 7f992e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:25.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.312+0000 7f992e7fc640 1 -- 172.21.15.170:0/1157040525 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9928004c30 con 0x7f99380a9d70 2024-03-29T11:20:25.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.312+0000 7f992f7fe640 4 set_mon_vals no callback set 2024-03-29T11:20:25.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.314+0000 7f99400a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:25.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.313+0000 7f992e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:25.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.314+0000 7f992e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:25.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.314+0000 7f992e7fc640 10 monclient: dump: 2024-03-29T11:20:25.295 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:25.295 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:25.295 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:25.295 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:25.295 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:25.295 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:25.295 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:25.295 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:25.295 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:25.295 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:25.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.314+0000 7f992e7fc640 1 -- 172.21.15.170:0/1157040525 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f992801ac00 con 0x7f99380a9d70 2024-03-29T11:20:25.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.315+0000 7f99400a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:25.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.315+0000 7f99400a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:20:25.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.315+0000 7f99400a5640 1 -- 172.21.15.170:0/1157040525 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9920005080 con 0x7f99380a9d70 2024-03-29T11:20:25.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.316+0000 7f993d640640 10 monclient: get_auth_request con 0x7f991c06dd80 auth_method 0 2024-03-29T11:20:25.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.316+0000 7f992e7fc640 1 -- 172.21.15.170:0/1157040525 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f9928084930 con 0x7f99380a9d70 2024-03-29T11:20:25.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.316+0000 7f992e7fc640 10 client.4504.objecter ms_dispatch 0x7f9938000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:25.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.316+0000 7f992e7fc640 3 client.4504.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:25.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.316+0000 7f992e7fc640 3 client.4504.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:25.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.316+0000 7f992e7fc640 20 client.4504.objecter dump_active .. 0 homeless 2024-03-29T11:20:25.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.316+0000 7f992e7fc640 10 client.4504.objecter ms_handle_connect 0x7f991c06dd80 2024-03-29T11:20:25.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.323+0000 7f992e7fc640 1 -- 172.21.15.170:0/1157040525 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9928050210 con 0x7f99380a9d70 2024-03-29T11:20:25.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.323+0000 7f992e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:25.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.323+0000 7f992e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:25.650 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.673+0000 7f99400a5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:25.650 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.673+0000 7f99400a5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:25.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.673+0000 7f99400a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:20:25.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.673+0000 7f99400a5640 1 -- 172.21.15.170:0/1157040525 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9920005330 con 0x7f99380a9d70 2024-03-29T11:20:25.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.674+0000 7f992e7fc640 1 -- 172.21.15.170:0/1157040525 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f9928052140 con 0x7f99380a9d70 2024-03-29T11:20:25.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.674+0000 7f992e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:25.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.674+0000 7f992e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:25.652 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:20:25.652 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:25.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.678+0000 7f99400a5640 10 librados: watch_flush enter 2024-03-29T11:20:25.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.678+0000 7f99400a5640 10 librados: watch_flush exit 2024-03-29T11:20:25.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.678+0000 7f99400a5640 20 client.4504.objecter shutdown clearing up homeless session... 2024-03-29T11:20:25.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.678+0000 7f99400a5640 10 client.4504.objecter successfully canceled tick 2024-03-29T11:20:25.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.678+0000 7f99400a5640 1 -- 172.21.15.170:0/1157040525 >> v1:172.21.15.170:6816/3298994444 conn(0x7f991c06dd80 legacy=0x7f991c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:25.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.678+0000 7f99400a5640 10 monclient: shutdown 2024-03-29T11:20:25.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.678+0000 7f99400a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:25.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.678+0000 7f99400a5640 1 -- 172.21.15.170:0/1157040525 >> v1:172.21.15.184:6789/0 conn(0x7f99380a9d70 legacy=0x7f99381dabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:25.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.678+0000 7f99400a5640 1 -- 172.21.15.170:0/1157040525 shutdown_connections 2024-03-29T11:20:25.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.678+0000 7f99400a5640 1 -- 172.21.15.170:0/1157040525 >> 172.21.15.170:0/1157040525 conn(0x7f9938066a00 msgr2=0x7f99380b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:25.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.679+0000 7f99400a5640 1 -- 172.21.15.170:0/1157040525 shutdown_connections 2024-03-29T11:20:25.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.679+0000 7f99400a5640 1 -- 172.21.15.170:0/1157040525 wait complete. 2024-03-29T11:20:25.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:25.679+0000 7f99400a5640 1 librados: shutdown 2024-03-29T11:20:25.670 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:32.479 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:32.630 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.654+0000 7f38a9c4a640 10 monclient: get_monmap_and_config 2024-03-29T11:20:32.630 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.654+0000 7f38a9c4a640 10 monclient: build_initial_monmap 2024-03-29T11:20:32.630 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.654+0000 7f38a9c4a640 10 monclient: monmap: 2024-03-29T11:20:32.630 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:20:32.630 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:32.631 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:20:32.654530+0000 2024-03-29T11:20:32.631 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:20:32.654530+0000 2024-03-29T11:20:32.631 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:32.631 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:32.631 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:20:32.631 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:20:32.631 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:20:32.631 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:32.634 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.658+0000 7f38a9c4a640 1 Processor -- start 2024-03-29T11:20:32.634 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.658+0000 7f38a9c4a640 1 -- start start 2024-03-29T11:20:32.634 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.658+0000 7f38a9c4a640 10 monclient: init 2024-03-29T11:20:32.638 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.659+0000 7f38a9c4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:32.638 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.659+0000 7f38a9c4a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:32.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.659+0000 7f38a9c4a640 10 monclient(hunting): picked mon.noname-a con 0x7f38a4074650 addr v1:172.21.15.170:6789/0 2024-03-29T11:20:32.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.659+0000 7f38a9c4a640 10 monclient(hunting): picked mon.noname-b con 0x7f38a4072e80 addr v1:172.21.15.170:6790/0 2024-03-29T11:20:32.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.659+0000 7f38a9c4a640 10 monclient(hunting): picked mon.noname-c con 0x7f38a406f510 addr v1:172.21.15.184:6789/0 2024-03-29T11:20:32.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.659+0000 7f38a9c4a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a406f080 con 0x7f38a4074650 2024-03-29T11:20:32.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.659+0000 7f38a9c4a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a4070900 con 0x7f38a406f510 2024-03-29T11:20:32.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.659+0000 7f38a9c4a640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a4143d10 con 0x7f38a4072e80 2024-03-29T11:20:32.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.659+0000 7f38a9c4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:32.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.659+0000 7f38a9c4a640 10 monclient(hunting): authenticate will time out at 1413.970344s 2024-03-29T11:20:32.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.662+0000 7f38a3fff640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f38a406f510 0x7f38a4151d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:60395/0 (socket says 172.21.15.170:60395) 2024-03-29T11:20:32.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.662+0000 7f38a3fff640 1 -- 172.21.15.170:0/1699359053 learned_addr learned my addr 172.21.15.170:0/1699359053 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:20:32.641 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.663+0000 7f38a27fc640 1 -- 172.21.15.170:0/1699359053 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584687323 0 0) 0x7f38a406f080 con 0x7f38a4074650 2024-03-29T11:20:32.641 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.663+0000 7f38a27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:32.641 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.663+0000 7f38a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:32.641 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.663+0000 7f38a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:32.641 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.663+0000 7f38a27fc640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:20:32.642 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.663+0000 7f38a27fc640 1 -- 172.21.15.170:0/1699359053 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f388c003520 con 0x7f38a4074650 2024-03-29T11:20:32.642 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.663+0000 7f38a27fc640 1 -- 172.21.15.170:0/1699359053 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483332084 0 0) 0x7f38a4070900 con 0x7f38a406f510 2024-03-29T11:20:32.642 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.663+0000 7f38a27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:32.642 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.663+0000 7f38a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:32.642 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.663+0000 7f38a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:32.642 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.663+0000 7f38a27fc640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:20:32.642 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.663+0000 7f38a27fc640 1 -- 172.21.15.170:0/1699359053 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a406f080 con 0x7f38a406f510 2024-03-29T11:20:32.642 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.663+0000 7f38a27fc640 1 -- 172.21.15.170:0/1699359053 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2187953863 0 0) 0x7f388c003520 con 0x7f38a4074650 2024-03-29T11:20:32.642 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.663+0000 7f38a27fc640 1 -- 172.21.15.170:0/1699359053 >> v1:172.21.15.170:6790/0 conn(0x7f38a4072e80 legacy=0x7f38a41499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:32.642 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a27fc640 1 -- 172.21.15.170:0/1699359053 >> v1:172.21.15.184:6789/0 conn(0x7f38a406f510 legacy=0x7f38a4151d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:32.642 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:32.642 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a27fc640 1 monclient: found mon.noname-a 2024-03-29T11:20:32.643 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:32.643 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:20:32.643 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a27fc640 1 -- 172.21.15.170:0/1699359053 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a406e8d0 con 0x7f38a4074650 2024-03-29T11:20:32.643 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a27fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:32.643 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:32.643 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:32.643 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a27fc640 1 -- 172.21.15.170:0/1699359053 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f389c001a10 con 0x7f38a4074650 2024-03-29T11:20:32.643 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:32.643 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:32.643 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a27fc640 10 monclient: dump: 2024-03-29T11:20:32.643 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:32.643 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:32.644 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:32.644 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:32.644 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:32.644 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:32.644 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:32.644 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:32.644 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:32.644 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:32.644 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a27fc640 1 -- 172.21.15.170:0/1699359053 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f389c001be0 con 0x7f38a4074650 2024-03-29T11:20:32.644 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:32.644 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a27fc640 1 -- 172.21.15.170:0/1699359053 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f389c001f70 con 0x7f38a4074650 2024-03-29T11:20:32.644 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:32.645 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:32.645 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a27fc640 10 monclient: dump: 2024-03-29T11:20:32.645 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:32.645 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:32.645 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:32.645 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:32.645 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:32.645 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:32.645 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:32.645 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:32.645 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:32.645 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:32.645 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a9c4a640 5 monclient: authenticate success, global_id 4539 2024-03-29T11:20:32.645 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a9c4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:32.646 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.664+0000 7f38a9c4a640 4 set_mon_vals no callback set 2024-03-29T11:20:32.646 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.665+0000 7f38a9c4a640 10 monclient: shutdown 2024-03-29T11:20:32.646 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.665+0000 7f38a9c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:32.646 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.665+0000 7f38a9c4a640 1 -- 172.21.15.170:0/1699359053 >> v1:172.21.15.170:6789/0 conn(0x7f38a4074650 legacy=0x7f38a4143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:32.646 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.665+0000 7f38a9c4a640 1 -- 172.21.15.170:0/1699359053 shutdown_connections 2024-03-29T11:20:32.646 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.665+0000 7f38a9c4a640 1 -- 172.21.15.170:0/1699359053 >> 172.21.15.170:0/1699359053 conn(0x7f38a40b4200 msgr2=0x7f38a40b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:32.647 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.665+0000 7f38a9c4a640 1 -- 172.21.15.170:0/1699359053 shutdown_connections 2024-03-29T11:20:32.648 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.666+0000 7f38a9c4a640 1 -- 172.21.15.170:0/1699359053 wait complete. 2024-03-29T11:20:32.648 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.667+0000 7f38a9c4a640 10 monclient: build_initial_monmap 2024-03-29T11:20:32.648 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.667+0000 7f38a9c4a640 10 monclient: monmap: 2024-03-29T11:20:32.648 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:20:32.648 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:32.648 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:20:32.648 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:20:32.648 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:32.648 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:32.648 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:20:32.648 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:20:32.648 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:20:32.648 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:32.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.668+0000 7f38a9c4a640 1 librados: starting msgr at 2024-03-29T11:20:32.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.668+0000 7f38a9c4a640 1 librados: starting objecter 2024-03-29T11:20:32.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.668+0000 7f38a9c4a640 1 Processor -- start 2024-03-29T11:20:32.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.669+0000 7f38a9c4a640 1 -- start start 2024-03-29T11:20:32.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.669+0000 7f38a9c4a640 1 librados: setting wanted keys 2024-03-29T11:20:32.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.669+0000 7f38a9c4a640 1 librados: calling monclient init 2024-03-29T11:20:32.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.669+0000 7f38a9c4a640 10 monclient: init 2024-03-29T11:20:32.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.670+0000 7f38a9c4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:32.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.670+0000 7f38a9c4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:32.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.670+0000 7f38a9c4a640 10 monclient(hunting): picked mon.noname-a con 0x7f38a406f510 addr v1:172.21.15.170:6789/0 2024-03-29T11:20:32.650 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.672+0000 7f38a37fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f38a406f510 0x7f38a4145cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:37522/0 (socket says 172.21.15.170:37522) 2024-03-29T11:20:32.650 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.672+0000 7f38a37fe640 1 -- 172.21.15.170:0/2810625303 learned_addr learned my addr 172.21.15.170:0/2810625303 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:20:32.650 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.672+0000 7f38a9c4a640 10 monclient(hunting): picked mon.noname-b con 0x7f38a4074650 addr v1:172.21.15.184:6789/0 2024-03-29T11:20:32.650 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.672+0000 7f38a9c4a640 10 monclient(hunting): picked mon.noname-c con 0x7f38a41e4b10 addr v1:172.21.15.170:6790/0 2024-03-29T11:20:32.650 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.672+0000 7f38a9c4a640 1 -- 172.21.15.170:0/2810625303 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a41e3f70 con 0x7f38a406f510 2024-03-29T11:20:32.650 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.672+0000 7f38a9c4a640 1 -- 172.21.15.170:0/2810625303 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a40a94e0 con 0x7f38a4074650 2024-03-29T11:20:32.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.672+0000 7f38a9c4a640 1 -- 172.21.15.170:0/2810625303 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a4145110 con 0x7f38a41e4b10 2024-03-29T11:20:32.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.672+0000 7f38a9c4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:32.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.672+0000 7f38a9c4a640 10 monclient(hunting): authenticate will time out at 1413.983259s 2024-03-29T11:20:32.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.672+0000 7f388bfff640 10 client.?.objecter ms_handle_connect 0x7f38a406f510 2024-03-29T11:20:32.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.672+0000 7f388bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:32.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.672+0000 7f388bfff640 1 -- 172.21.15.170:0/2810625303 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1586391206 0 0) 0x7f38a41e3f70 con 0x7f38a406f510 2024-03-29T11:20:32.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.672+0000 7f388bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:32.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.672+0000 7f388bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:32.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.672+0000 7f388bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:32.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.672+0000 7f388bfff640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:20:32.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.672+0000 7f388bfff640 1 -- 172.21.15.170:0/2810625303 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3890004eb0 con 0x7f38a406f510 2024-03-29T11:20:32.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.672+0000 7f388bfff640 1 -- 172.21.15.170:0/2810625303 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3415588331 0 0) 0x7f3890004eb0 con 0x7f38a406f510 2024-03-29T11:20:32.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f388bfff640 1 -- 172.21.15.170:0/2810625303 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38a41e3f70 con 0x7f38a406f510 2024-03-29T11:20:32.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f388bfff640 1 -- 172.21.15.170:0/2810625303 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f389c001f30 con 0x7f38a406f510 2024-03-29T11:20:32.652 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f388bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:32.652 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f388bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:32.652 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f388bfff640 10 monclient(hunting): dump: 2024-03-29T11:20:32.652 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:32.652 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:32.652 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:32.652 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:32.652 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:32.652 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:32.652 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:32.652 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:32.652 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:32.652 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:32.652 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f388bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:32.652 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f388bfff640 1 -- 172.21.15.170:0/2810625303 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1026486481 0 0) 0x7f38a41e3f70 con 0x7f38a406f510 2024-03-29T11:20:32.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f388bfff640 1 -- 172.21.15.170:0/2810625303 >> v1:172.21.15.170:6790/0 conn(0x7f38a41e4b10 legacy=0x7f38a41ee880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:32.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f388bfff640 1 -- 172.21.15.170:0/2810625303 >> v1:172.21.15.184:6789/0 conn(0x7f38a4074650 legacy=0x7f38a41e4400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:32.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f388bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:32.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f388bfff640 1 monclient: found mon.a 2024-03-29T11:20:32.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f388bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:32.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f388bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:20:32.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f388bfff640 1 -- 172.21.15.170:0/2810625303 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a41452a0 con 0x7f38a406f510 2024-03-29T11:20:32.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f388bfff640 10 monclient: _finish_auth 0 2024-03-29T11:20:32.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f388bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:32.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f388bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:32.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f38a9c4a640 5 monclient: authenticate success, global_id 4542 2024-03-29T11:20:32.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f38a9c4a640 10 monclient: _renew_subs 2024-03-29T11:20:32.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f38a9c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:20:32.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f38a9c4a640 1 -- 172.21.15.170:0/2810625303 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a41463c0 con 0x7f38a406f510 2024-03-29T11:20:32.654 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f38a9c4a640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:32.654 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f38a9c4a640 10 monclient: _renew_subs 2024-03-29T11:20:32.654 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f38a9c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:20:32.654 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f38a9c4a640 1 -- 172.21.15.170:0/2810625303 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a41d5950 con 0x7f38a406f510 2024-03-29T11:20:32.654 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.673+0000 7f38a9c4a640 1 librados: init done 2024-03-29T11:20:32.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.676+0000 7f38a9c4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:32.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.676+0000 7f38a9c4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:32.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.676+0000 7f38a9c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:20:32.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.676+0000 7f38a9c4a640 1 -- 172.21.15.170:0/2810625303 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a4067bf0 con 0x7f38a406f510 2024-03-29T11:20:32.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.678+0000 7f388bfff640 1 -- 172.21.15.170:0/2810625303 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f389c003c20 con 0x7f38a406f510 2024-03-29T11:20:32.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.678+0000 7f388bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:32.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.678+0000 7f388bfff640 1 -- 172.21.15.170:0/2810625303 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f389c003e10 con 0x7f38a406f510 2024-03-29T11:20:32.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.678+0000 7f388bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:32.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.678+0000 7f388bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:32.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.678+0000 7f388bfff640 10 monclient: dump: 2024-03-29T11:20:32.656 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:32.656 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:32.656 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:32.656 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:32.656 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:32.656 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:32.656 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:32.656 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:32.656 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:32.656 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:32.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.678+0000 7f38a1ffb640 4 set_mon_vals no callback set 2024-03-29T11:20:32.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.679+0000 7f388bfff640 1 -- 172.21.15.170:0/2810625303 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f389c004030 con 0x7f38a406f510 2024-03-29T11:20:32.657 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.679+0000 7f388bfff640 1 -- 172.21.15.170:0/2810625303 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f389c084d00 con 0x7f38a406f510 2024-03-29T11:20:32.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.679+0000 7f388bfff640 10 client.4542.objecter ms_dispatch 0x7f38a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:32.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.679+0000 7f388bfff640 3 client.4542.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:32.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.679+0000 7f388bfff640 3 client.4542.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:32.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.679+0000 7f38a2ffd640 10 monclient: get_auth_request con 0x7f389006dc90 auth_method 0 2024-03-29T11:20:32.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.680+0000 7f388bfff640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-29T11:20:32.658 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.680+0000 7f388bfff640 10 client.4542.objecter ms_handle_connect 0x7f389006dc90 2024-03-29T11:20:32.662 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.686+0000 7f388bfff640 1 -- 172.21.15.170:0/2810625303 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f389c0505e0 con 0x7f38a406f510 2024-03-29T11:20:32.663 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.686+0000 7f388bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:32.663 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:32.686+0000 7f388bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:32.987 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.011+0000 7f38a9c4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:32.987 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.011+0000 7f38a9c4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:32.987 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.011+0000 7f38a9c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:20:32.987 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.011+0000 7f38a9c4a640 1 -- 172.21.15.170:0/2810625303 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38a4075f10 con 0x7f38a406f510 2024-03-29T11:20:32.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.012+0000 7f388bfff640 1 -- 172.21.15.170:0/2810625303 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f389c052510 con 0x7f38a406f510 2024-03-29T11:20:32.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.012+0000 7f388bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:32.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.012+0000 7f388bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:32.989 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:20:32.989 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:32.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.016+0000 7f38a9c4a640 10 librados: watch_flush enter 2024-03-29T11:20:32.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.016+0000 7f38a9c4a640 10 librados: watch_flush exit 2024-03-29T11:20:32.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.016+0000 7f38a9c4a640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-29T11:20:32.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.016+0000 7f38a9c4a640 10 client.4542.objecter successfully canceled tick 2024-03-29T11:20:32.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.016+0000 7f38a9c4a640 1 -- 172.21.15.170:0/2810625303 >> v1:172.21.15.170:6816/3298994444 conn(0x7f389006dc90 legacy=0x7f3890070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:32.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.016+0000 7f38a9c4a640 10 monclient: shutdown 2024-03-29T11:20:32.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.016+0000 7f38a9c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:32.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.016+0000 7f38a9c4a640 1 -- 172.21.15.170:0/2810625303 >> v1:172.21.15.170:6789/0 conn(0x7f38a406f510 legacy=0x7f38a4145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:32.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.016+0000 7f38a9c4a640 1 -- 172.21.15.170:0/2810625303 shutdown_connections 2024-03-29T11:20:32.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.016+0000 7f38a9c4a640 1 -- 172.21.15.170:0/2810625303 >> 172.21.15.170:0/2810625303 conn(0x7f38a40b4200 msgr2=0x7f38a40b1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:32.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.017+0000 7f38a9c4a640 1 -- 172.21.15.170:0/2810625303 shutdown_connections 2024-03-29T11:20:32.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.017+0000 7f38a9c4a640 1 -- 172.21.15.170:0/2810625303 wait complete. 2024-03-29T11:20:32.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:33.017+0000 7f38a9c4a640 1 librados: shutdown 2024-03-29T11:20:33.010 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:39.816 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:39.964 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.987+0000 7f9a82754640 10 monclient: get_monmap_and_config 2024-03-29T11:20:39.964 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.987+0000 7f9a82754640 10 monclient: build_initial_monmap 2024-03-29T11:20:39.964 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.987+0000 7f9a82754640 10 monclient: monmap: 2024-03-29T11:20:39.964 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:20:39.964 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:39.964 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:20:39.988166+0000 2024-03-29T11:20:39.964 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:20:39.988166+0000 2024-03-29T11:20:39.964 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:39.964 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:39.965 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:20:39.965 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:20:39.965 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:20:39.965 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:39.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.988+0000 7f9a82754640 1 Processor -- start 2024-03-29T11:20:39.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.988+0000 7f9a82754640 1 -- start start 2024-03-29T11:20:39.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.988+0000 7f9a82754640 10 monclient: init 2024-03-29T11:20:39.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.989+0000 7f9a82754640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:39.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.989+0000 7f9a82754640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:39.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.989+0000 7f9a82754640 10 monclient(hunting): picked mon.noname-b con 0x7f9a7c1457f0 addr v1:172.21.15.170:6790/0 2024-03-29T11:20:39.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.989+0000 7f9a82754640 10 monclient(hunting): picked mon.noname-a con 0x7f9a7c13f650 addr v1:172.21.15.170:6789/0 2024-03-29T11:20:39.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.989+0000 7f9a82754640 10 monclient(hunting): picked mon.noname-c con 0x7f9a7c0a9d70 addr v1:172.21.15.184:6789/0 2024-03-29T11:20:39.970 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.989+0000 7f9a82754640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7c0aaef0 con 0x7f9a7c13f650 2024-03-29T11:20:39.970 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.989+0000 7f9a82754640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7c145bd0 con 0x7f9a7c0a9d70 2024-03-29T11:20:39.970 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.989+0000 7f9a82754640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7c145eb0 con 0x7f9a7c1457f0 2024-03-29T11:20:39.970 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.989+0000 7f9a82754640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:39.971 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.989+0000 7f9a82754640 10 monclient(hunting): authenticate will time out at 1421.300902s 2024-03-29T11:20:39.971 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.993+0000 7f9a80cf1640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f9a7c0a9d70 0x7f9a7c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:50755/0 (socket says 172.21.15.170:50755) 2024-03-29T11:20:39.971 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.993+0000 7f9a80cf1640 1 -- 172.21.15.170:0/2444987858 learned_addr learned my addr 172.21.15.170:0/2444987858 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:20:39.974 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.994+0000 7f9a7affd640 1 -- 172.21.15.170:0/2444987858 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936397807 0 0) 0x7f9a7c145eb0 con 0x7f9a7c1457f0 2024-03-29T11:20:39.974 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.994+0000 7f9a7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:39.974 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.994+0000 7f9a7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:39.974 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.994+0000 7f9a7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:39.974 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.994+0000 7f9a7affd640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:20:39.974 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.994+0000 7f9a7affd640 1 -- 172.21.15.170:0/2444987858 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a64003520 con 0x7f9a7c1457f0 2024-03-29T11:20:39.974 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.994+0000 7f9a7affd640 1 -- 172.21.15.170:0/2444987858 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3346402472 0 0) 0x7f9a7c0aaef0 con 0x7f9a7c13f650 2024-03-29T11:20:39.974 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.994+0000 7f9a7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:39.974 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.994+0000 7f9a7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:39.974 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.994+0000 7f9a7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:39.974 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.994+0000 7f9a7affd640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:20:39.974 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.994+0000 7f9a7affd640 1 -- 172.21.15.170:0/2444987858 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a7c145eb0 con 0x7f9a7c13f650 2024-03-29T11:20:39.974 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.995+0000 7f9a7affd640 1 -- 172.21.15.170:0/2444987858 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1711294050 0 0) 0x7f9a7c145eb0 con 0x7f9a7c13f650 2024-03-29T11:20:39.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.995+0000 7f9a7affd640 1 -- 172.21.15.170:0/2444987858 >> v1:172.21.15.170:6790/0 conn(0x7f9a7c1457f0 legacy=0x7f9a7c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:39.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.995+0000 7f9a7affd640 1 -- 172.21.15.170:0/2444987858 >> v1:172.21.15.184:6789/0 conn(0x7f9a7c0a9d70 legacy=0x7f9a7c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:39.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.997+0000 7f9a7affd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:39.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.997+0000 7f9a7affd640 1 monclient: found mon.noname-a 2024-03-29T11:20:39.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.997+0000 7f9a7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:39.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.997+0000 7f9a7affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:20:39.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.997+0000 7f9a7affd640 1 -- 172.21.15.170:0/2444987858 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c146340 con 0x7f9a7c13f650 2024-03-29T11:20:39.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.997+0000 7f9a7affd640 10 monclient: _finish_auth 0 2024-03-29T11:20:39.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.997+0000 7f9a7affd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:39.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.997+0000 7f9a7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:39.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.997+0000 7f9a7affd640 1 -- 172.21.15.170:0/2444987858 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9a700015a0 con 0x7f9a7c13f650 2024-03-29T11:20:39.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.997+0000 7f9a7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:39.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.997+0000 7f9a7affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:39.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.997+0000 7f9a7affd640 10 monclient: dump: 2024-03-29T11:20:39.975 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:39.976 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:39.976 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:39.976 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:39.976 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:39.976 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:39.976 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:39.976 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:39.976 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:39.976 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:39.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.997+0000 7f9a82754640 5 monclient: authenticate success, global_id 4545 2024-03-29T11:20:39.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.997+0000 7f9a82754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:39.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.998+0000 7f9a7affd640 1 -- 172.21.15.170:0/2444987858 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a70001700 con 0x7f9a7c13f650 2024-03-29T11:20:39.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.998+0000 7f9a7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:39.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.998+0000 7f9a7affd640 1 -- 172.21.15.170:0/2444987858 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9a70001ad0 con 0x7f9a7c13f650 2024-03-29T11:20:39.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.998+0000 7f9a7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:39.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.998+0000 7f9a7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:39.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.998+0000 7f9a7affd640 10 monclient: dump: 2024-03-29T11:20:39.977 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:39.977 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:39.977 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:39.977 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:39.978 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:39.978 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:39.978 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:39.978 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:39.978 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:39.978 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:39.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.998+0000 7f9a82754640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:39.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.998+0000 7f9a82754640 4 set_mon_vals no callback set 2024-03-29T11:20:39.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.999+0000 7f9a82754640 10 monclient: shutdown 2024-03-29T11:20:39.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.999+0000 7f9a82754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:39.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.999+0000 7f9a82754640 1 -- 172.21.15.170:0/2444987858 >> v1:172.21.15.170:6789/0 conn(0x7f9a7c13f650 legacy=0x7f9a7c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:39.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.999+0000 7f9a82754640 1 -- 172.21.15.170:0/2444987858 shutdown_connections 2024-03-29T11:20:39.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:39.999+0000 7f9a82754640 1 -- 172.21.15.170:0/2444987858 >> 172.21.15.170:0/2444987858 conn(0x7f9a7c066a00 msgr2=0x7f9a7c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:39.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.000+0000 7f9a82754640 1 -- 172.21.15.170:0/2444987858 shutdown_connections 2024-03-29T11:20:39.979 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.000+0000 7f9a82754640 1 -- 172.21.15.170:0/2444987858 wait complete. 2024-03-29T11:20:39.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.001+0000 7f9a82754640 10 monclient: build_initial_monmap 2024-03-29T11:20:39.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.001+0000 7f9a82754640 10 monclient: monmap: 2024-03-29T11:20:39.981 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:20:39.981 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:39.981 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:20:39.981 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:20:39.981 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:39.981 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:39.981 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:20:39.981 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:20:39.981 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:20:39.981 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:39.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.001+0000 7f9a82754640 1 librados: starting msgr at 2024-03-29T11:20:39.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.001+0000 7f9a82754640 1 librados: starting objecter 2024-03-29T11:20:39.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.001+0000 7f9a82754640 1 Processor -- start 2024-03-29T11:20:39.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.002+0000 7f9a82754640 1 -- start start 2024-03-29T11:20:39.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.002+0000 7f9a82754640 1 librados: setting wanted keys 2024-03-29T11:20:39.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.002+0000 7f9a82754640 1 librados: calling monclient init 2024-03-29T11:20:39.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.002+0000 7f9a82754640 10 monclient: init 2024-03-29T11:20:39.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.003+0000 7f9a82754640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:39.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.003+0000 7f9a82754640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:39.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.003+0000 7f9a82754640 10 monclient(hunting): picked mon.noname-a con 0x7f9a7c0a9d70 addr v1:172.21.15.170:6789/0 2024-03-29T11:20:39.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.003+0000 7f9a82754640 10 monclient(hunting): picked mon.noname-c con 0x7f9a7c13f650 addr v1:172.21.15.170:6790/0 2024-03-29T11:20:39.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.003+0000 7f9a82754640 10 monclient(hunting): picked mon.noname-b con 0x7f9a7c1457f0 addr v1:172.21.15.184:6789/0 2024-03-29T11:20:39.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.003+0000 7f9a82754640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7c145350 con 0x7f9a7c0a9d70 2024-03-29T11:20:39.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.003+0000 7f9a82754640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7c142940 con 0x7f9a7c1457f0 2024-03-29T11:20:39.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.003+0000 7f9a82754640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7c1417d0 con 0x7f9a7c13f650 2024-03-29T11:20:39.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.003+0000 7f9a82754640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:39.983 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.003+0000 7f9a82754640 10 monclient(hunting): authenticate will time out at 1421.314512s 2024-03-29T11:20:39.983 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.005+0000 7f9a7b7fe640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f9a7c13f650 0x7f9a7c1db6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:38180/0 (socket says 172.21.15.170:38180) 2024-03-29T11:20:39.983 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.005+0000 7f9a7b7fe640 1 -- 172.21.15.170:0/4070288126 learned_addr learned my addr 172.21.15.170:0/4070288126 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:20:39.987 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.008+0000 7f9a78ff9640 10 client.?.objecter ms_handle_connect 0x7f9a7c13f650 2024-03-29T11:20:39.987 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.008+0000 7f9a78ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:39.987 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.008+0000 7f9a78ff9640 1 -- 172.21.15.170:0/4070288126 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682118385 0 0) 0x7f9a7c1417d0 con 0x7f9a7c13f650 2024-03-29T11:20:39.987 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.008+0000 7f9a78ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:39.987 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.008+0000 7f9a78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:39.987 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.008+0000 7f9a78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:39.987 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.008+0000 7f9a78ff9640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:20:39.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.009+0000 7f9a78ff9640 1 -- 172.21.15.170:0/4070288126 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a68004eb0 con 0x7f9a7c13f650 2024-03-29T11:20:39.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.009+0000 7f9a78ff9640 10 client.?.objecter ms_handle_connect 0x7f9a7c1457f0 2024-03-29T11:20:39.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.009+0000 7f9a78ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:39.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.009+0000 7f9a78ff9640 1 -- 172.21.15.170:0/4070288126 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287715509 0 0) 0x7f9a7c142940 con 0x7f9a7c1457f0 2024-03-29T11:20:39.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.009+0000 7f9a78ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:39.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.009+0000 7f9a78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:39.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.009+0000 7f9a78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:39.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.009+0000 7f9a78ff9640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:20:39.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.009+0000 7f9a78ff9640 1 -- 172.21.15.170:0/4070288126 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a7c1417d0 con 0x7f9a7c1457f0 2024-03-29T11:20:39.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.009+0000 7f9a78ff9640 1 -- 172.21.15.170:0/4070288126 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3298663324 0 0) 0x7f9a68004eb0 con 0x7f9a7c13f650 2024-03-29T11:20:39.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.009+0000 7f9a78ff9640 1 -- 172.21.15.170:0/4070288126 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a7c142940 con 0x7f9a7c13f650 2024-03-29T11:20:39.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.009+0000 7f9a78ff9640 1 -- 172.21.15.170:0/4070288126 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9a70001b10 con 0x7f9a7c13f650 2024-03-29T11:20:39.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.009+0000 7f9a78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:39.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a78ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:39.989 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a78ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:39.989 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:39.989 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:39.989 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:39.989 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:39.989 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:39.989 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:39.989 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:39.989 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:39.989 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:39.989 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:39.989 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a78ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:39.989 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a78ff9640 10 client.?.objecter ms_handle_connect 0x7f9a7c0a9d70 2024-03-29T11:20:39.989 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a78ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:39.989 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a78ff9640 1 -- 172.21.15.170:0/4070288126 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 262241476 0 0) 0x7f9a7c142940 con 0x7f9a7c13f650 2024-03-29T11:20:39.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a78ff9640 1 -- 172.21.15.170:0/4070288126 >> v1:172.21.15.184:6789/0 conn(0x7f9a7c1457f0 legacy=0x7f9a7c1e6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:39.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a78ff9640 1 -- 172.21.15.170:0/4070288126 >> v1:172.21.15.170:6789/0 conn(0x7f9a7c0a9d70 legacy=0x7f9a7c0715d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:39.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a78ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:20:39.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a78ff9640 1 monclient: found mon.c 2024-03-29T11:20:39.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a78ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:39.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a78ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:20:39.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a78ff9640 1 -- 172.21.15.170:0/4070288126 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c141960 con 0x7f9a7c13f650 2024-03-29T11:20:39.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a78ff9640 10 monclient: _finish_auth 0 2024-03-29T11:20:39.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a78ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:20:39.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:39.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a82754640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:20:39.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a82754640 10 monclient: _renew_subs 2024-03-29T11:20:39.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a82754640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:20:39.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a82754640 1 -- 172.21.15.170:0/4070288126 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a7c144710 con 0x7f9a7c13f650 2024-03-29T11:20:39.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a82754640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:39.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a82754640 10 monclient: _renew_subs 2024-03-29T11:20:39.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a82754640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:20:39.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a82754640 1 -- 172.21.15.170:0/4070288126 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a7c071ec0 con 0x7f9a7c13f650 2024-03-29T11:20:39.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.010+0000 7f9a82754640 1 librados: init done 2024-03-29T11:20:39.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.011+0000 7f9a78ff9640 1 -- 172.21.15.170:0/4070288126 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a700038a0 con 0x7f9a7c13f650 2024-03-29T11:20:39.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.011+0000 7f9a78ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:39.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.011+0000 7f9a78ff9640 1 -- 172.21.15.170:0/4070288126 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9a70003c70 con 0x7f9a7c13f650 2024-03-29T11:20:39.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.011+0000 7f9a78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:39.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.011+0000 7f9a78ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:39.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.011+0000 7f9a78ff9640 10 monclient: dump: 2024-03-29T11:20:39.991 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:39.991 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:39.991 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:39.991 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:39.991 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:39.991 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:39.992 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:39.992 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:39.992 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:39.992 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:39.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.012+0000 7f9a82754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:39.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.012+0000 7f9a82754640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:39.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.012+0000 7f9a82754640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:20:39.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.013+0000 7f9a78ff9640 1 -- 172.21.15.170:0/4070288126 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f9a70003e50 con 0x7f9a7c13f650 2024-03-29T11:20:39.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.012+0000 7f9a82754640 1 -- 172.21.15.170:0/4070288126 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7c144ab0 con 0x7f9a7c13f650 2024-03-29T11:20:39.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.014+0000 7f9a78ff9640 1 -- 172.21.15.170:0/4070288126 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f9a70084c60 con 0x7f9a7c13f650 2024-03-29T11:20:39.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.014+0000 7f9a78ff9640 10 client.4589.objecter ms_dispatch 0x7f9a7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:39.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.014+0000 7f9a78ff9640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:39.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.014+0000 7f9a78ff9640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:39.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.014+0000 7f9a7bfff640 10 monclient: get_auth_request con 0x7f9a6806db80 auth_method 0 2024-03-29T11:20:39.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.014+0000 7f9a78ff9640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-29T11:20:39.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.014+0000 7f9a7a7fc640 4 set_mon_vals no callback set 2024-03-29T11:20:39.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.014+0000 7f9a78ff9640 10 client.4589.objecter ms_handle_connect 0x7f9a6806db80 2024-03-29T11:20:40.001 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.025+0000 7f9a78ff9640 1 -- 172.21.15.170:0/4070288126 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a70050540 con 0x7f9a7c13f650 2024-03-29T11:20:40.001 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.025+0000 7f9a78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:40.001 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.025+0000 7f9a78ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:40.320 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.344+0000 7f9a82754640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:40.320 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.344+0000 7f9a82754640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:40.320 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.344+0000 7f9a82754640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:20:40.320 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.344+0000 7f9a82754640 1 -- 172.21.15.170:0/4070288126 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a7c142ad0 con 0x7f9a7c13f650 2024-03-29T11:20:40.321 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.345+0000 7f9a78ff9640 1 -- 172.21.15.170:0/4070288126 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f9a70052470 con 0x7f9a7c13f650 2024-03-29T11:20:40.321 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.345+0000 7f9a78ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:40.321 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.345+0000 7f9a78ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:40.321 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:20:40.322 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:40.324 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.348+0000 7f9a82754640 10 librados: watch_flush enter 2024-03-29T11:20:40.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.349+0000 7f9a82754640 10 librados: watch_flush exit 2024-03-29T11:20:40.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.349+0000 7f9a82754640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-29T11:20:40.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.349+0000 7f9a82754640 10 client.4589.objecter successfully canceled tick 2024-03-29T11:20:40.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.350+0000 7f9a82754640 1 -- 172.21.15.170:0/4070288126 >> v1:172.21.15.170:6816/3298994444 conn(0x7f9a6806db80 legacy=0x7f9a68070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:40.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.350+0000 7f9a82754640 10 monclient: shutdown 2024-03-29T11:20:40.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.350+0000 7f9a82754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:40.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.350+0000 7f9a82754640 1 -- 172.21.15.170:0/4070288126 >> v1:172.21.15.170:6790/0 conn(0x7f9a7c13f650 legacy=0x7f9a7c1db6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:40.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.351+0000 7f9a82754640 1 -- 172.21.15.170:0/4070288126 shutdown_connections 2024-03-29T11:20:40.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.351+0000 7f9a82754640 1 -- 172.21.15.170:0/4070288126 >> 172.21.15.170:0/4070288126 conn(0x7f9a7c066a00 msgr2=0x7f9a7c13c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:40.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.352+0000 7f9a82754640 1 -- 172.21.15.170:0/4070288126 shutdown_connections 2024-03-29T11:20:40.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.352+0000 7f9a82754640 1 -- 172.21.15.170:0/4070288126 wait complete. 2024-03-29T11:20:40.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:40.352+0000 7f9a82754640 1 librados: shutdown 2024-03-29T11:20:40.346 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:47.154 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:47.302 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.326+0000 7f9f020f1640 10 monclient: get_monmap_and_config 2024-03-29T11:20:47.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.326+0000 7f9f020f1640 10 monclient: build_initial_monmap 2024-03-29T11:20:47.303 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.326+0000 7f9f020f1640 10 monclient: monmap: 2024-03-29T11:20:47.303 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:20:47.303 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:47.303 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:20:47.327177+0000 2024-03-29T11:20:47.303 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:20:47.327177+0000 2024-03-29T11:20:47.303 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:47.303 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:47.303 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:20:47.303 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:20:47.303 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:20:47.303 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:47.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.327+0000 7f9f020f1640 1 Processor -- start 2024-03-29T11:20:47.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.328+0000 7f9f020f1640 1 -- start start 2024-03-29T11:20:47.304 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.328+0000 7f9f020f1640 10 monclient: init 2024-03-29T11:20:47.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.328+0000 7f9f020f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:47.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.328+0000 7f9f020f1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:47.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.329+0000 7f9f020f1640 10 monclient(hunting): picked mon.noname-b con 0x7f9efc0b1f60 addr v1:172.21.15.170:6790/0 2024-03-29T11:20:47.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.329+0000 7f9f020f1640 10 monclient(hunting): picked mon.noname-a con 0x7f9efc071eb0 addr v1:172.21.15.170:6789/0 2024-03-29T11:20:47.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.329+0000 7f9f020f1640 10 monclient(hunting): picked mon.noname-c con 0x7f9efc143d10 addr v1:172.21.15.184:6789/0 2024-03-29T11:20:47.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.329+0000 7f9f020f1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc06ef80 con 0x7f9efc071eb0 2024-03-29T11:20:47.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.329+0000 7f9f020f1640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc06f260 con 0x7f9efc143d10 2024-03-29T11:20:47.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.329+0000 7f9f020f1640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc0b40c0 con 0x7f9efc0b1f60 2024-03-29T11:20:47.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.329+0000 7f9f020f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:47.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.329+0000 7f9f020f1640 10 monclient(hunting): authenticate will time out at 1428.640113s 2024-03-29T11:20:47.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.329+0000 7f9efaffd640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f9efc071eb0 0x7f9efc146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:36884/0 (socket says 172.21.15.170:36884) 2024-03-29T11:20:47.308 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.329+0000 7f9efaffd640 1 -- 172.21.15.170:0/3311000851 learned_addr learned my addr 172.21.15.170:0/3311000851 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:20:47.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 1 -- 172.21.15.170:0/3311000851 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525285065 0 0) 0x7f9efc06f260 con 0x7f9efc143d10 2024-03-29T11:20:47.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:47.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:47.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:47.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:20:47.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 1 -- 172.21.15.170:0/3311000851 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee4003520 con 0x7f9efc143d10 2024-03-29T11:20:47.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 1 -- 172.21.15.170:0/3311000851 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691129269 0 0) 0x7f9efc06ef80 con 0x7f9efc071eb0 2024-03-29T11:20:47.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:47.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:47.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:47.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:20:47.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 1 -- 172.21.15.170:0/3311000851 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9efc06f260 con 0x7f9efc071eb0 2024-03-29T11:20:47.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 1 -- 172.21.15.170:0/3311000851 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 325162716 0 0) 0x7f9ee4003520 con 0x7f9efc143d10 2024-03-29T11:20:47.309 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 1 -- 172.21.15.170:0/3311000851 >> v1:172.21.15.170:6790/0 conn(0x7f9efc0b1f60 legacy=0x7f9efc143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:47.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 1 -- 172.21.15.170:0/3311000851 >> v1:172.21.15.170:6789/0 conn(0x7f9efc071eb0 legacy=0x7f9efc146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:47.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:47.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 1 monclient: found mon.noname-c 2024-03-29T11:20:47.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:47.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:20:47.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 1 -- 172.21.15.170:0/3311000851 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc155d70 con 0x7f9efc143d10 2024-03-29T11:20:47.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:47.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:47.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:47.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 1 -- 172.21.15.170:0/3311000851 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9ef4001a10 con 0x7f9efc143d10 2024-03-29T11:20:47.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:47.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:47.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 10 monclient: dump: 2024-03-29T11:20:47.310 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:47.311 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:47.311 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:47.311 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:47.311 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:47.311 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:47.311 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:47.311 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:47.311 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:47.311 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:47.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9f020f1640 5 monclient: authenticate success, global_id 4516 2024-03-29T11:20:47.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9f020f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:47.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 1 -- 172.21.15.170:0/3311000851 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ef4001b70 con 0x7f9efc143d10 2024-03-29T11:20:47.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:47.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 1 -- 172.21.15.170:0/3311000851 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9ef4001fc0 con 0x7f9efc143d10 2024-03-29T11:20:47.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:47.312 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:47.312 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9efa7fc640 10 monclient: dump: 2024-03-29T11:20:47.312 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:47.312 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:47.312 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:47.312 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:47.312 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:47.312 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:47.312 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:47.312 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:47.312 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:47.312 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:47.312 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9f020f1640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:47.312 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.330+0000 7f9f020f1640 4 set_mon_vals no callback set 2024-03-29T11:20:47.312 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.331+0000 7f9f020f1640 10 monclient: shutdown 2024-03-29T11:20:47.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.331+0000 7f9f020f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:47.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.331+0000 7f9f020f1640 1 -- 172.21.15.170:0/3311000851 >> v1:172.21.15.184:6789/0 conn(0x7f9efc143d10 legacy=0x7f9efc146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:47.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.331+0000 7f9f020f1640 1 -- 172.21.15.170:0/3311000851 shutdown_connections 2024-03-29T11:20:47.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.331+0000 7f9f020f1640 1 -- 172.21.15.170:0/3311000851 >> 172.21.15.170:0/3311000851 conn(0x7f9efc066a00 msgr2=0x7f9efc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:47.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.332+0000 7f9f020f1640 1 -- 172.21.15.170:0/3311000851 shutdown_connections 2024-03-29T11:20:47.313 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.332+0000 7f9f020f1640 1 -- 172.21.15.170:0/3311000851 wait complete. 2024-03-29T11:20:47.321 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.333+0000 7f9f020f1640 10 monclient: build_initial_monmap 2024-03-29T11:20:47.321 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.333+0000 7f9f020f1640 10 monclient: monmap: 2024-03-29T11:20:47.321 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:20:47.321 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:47.321 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:20:47.321 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:20:47.321 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:47.321 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:47.321 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:20:47.321 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:20:47.321 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:20:47.321 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:47.321 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.333+0000 7f9f020f1640 1 librados: starting msgr at 2024-03-29T11:20:47.321 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.333+0000 7f9f020f1640 1 librados: starting objecter 2024-03-29T11:20:47.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.333+0000 7f9f020f1640 1 Processor -- start 2024-03-29T11:20:47.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.334+0000 7f9f020f1640 1 -- start start 2024-03-29T11:20:47.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.334+0000 7f9f020f1640 1 librados: setting wanted keys 2024-03-29T11:20:47.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.334+0000 7f9f020f1640 1 librados: calling monclient init 2024-03-29T11:20:47.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.334+0000 7f9f020f1640 10 monclient: init 2024-03-29T11:20:47.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.334+0000 7f9f020f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:47.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.335+0000 7f9f020f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:47.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.335+0000 7f9f020f1640 10 monclient(hunting): picked mon.noname-b con 0x7f9efc0b1f60 addr v1:172.21.15.184:6789/0 2024-03-29T11:20:47.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.335+0000 7f9f020f1640 10 monclient(hunting): picked mon.noname-a con 0x7f9efc143d10 addr v1:172.21.15.170:6789/0 2024-03-29T11:20:47.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.335+0000 7f9efb7fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f9efc0b1f60 0x7f9efc147790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:34292/0 (socket says 172.21.15.170:34292) 2024-03-29T11:20:47.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.335+0000 7f9efb7fe640 1 -- 172.21.15.170:0/2736435923 learned_addr learned my addr 172.21.15.170:0/2736435923 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:20:47.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.335+0000 7f9f020f1640 10 monclient(hunting): picked mon.noname-c con 0x7f9efc149460 addr v1:172.21.15.170:6790/0 2024-03-29T11:20:47.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.335+0000 7f9f020f1640 1 -- 172.21.15.170:0/2736435923 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc14b0d0 con 0x7f9efc143d10 2024-03-29T11:20:47.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.335+0000 7f9f020f1640 1 -- 172.21.15.170:0/2736435923 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc13bb00 con 0x7f9efc0b1f60 2024-03-29T11:20:47.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.335+0000 7f9f020f1640 1 -- 172.21.15.170:0/2736435923 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc14b300 con 0x7f9efc149460 2024-03-29T11:20:47.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.335+0000 7f9ee3fff640 10 client.?.objecter ms_handle_connect 0x7f9efc0b1f60 2024-03-29T11:20:47.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.335+0000 7f9ee3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:47.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.335+0000 7f9ee3fff640 10 client.?.objecter ms_handle_connect 0x7f9efc149460 2024-03-29T11:20:47.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.335+0000 7f9ee3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:47.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.335+0000 7f9f020f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:47.324 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.335+0000 7f9f020f1640 10 monclient(hunting): authenticate will time out at 1428.646620s 2024-03-29T11:20:47.324 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279294594 0 0) 0x7f9efc13bb00 con 0x7f9efc0b1f60 2024-03-29T11:20:47.324 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:47.324 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:47.324 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:47.324 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:20:47.324 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee8004eb0 con 0x7f9efc0b1f60 2024-03-29T11:20:47.324 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 10 client.?.objecter ms_handle_connect 0x7f9efc143d10 2024-03-29T11:20:47.324 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:47.324 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164182153 0 0) 0x7f9efc14b300 con 0x7f9efc149460 2024-03-29T11:20:47.324 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:47.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:47.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:47.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:20:47.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9efc13bb00 con 0x7f9efc149460 2024-03-29T11:20:47.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 349419240 0 0) 0x7f9ee8004eb0 con 0x7f9efc0b1f60 2024-03-29T11:20:47.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9efc14b300 con 0x7f9efc0b1f60 2024-03-29T11:20:47.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9eec001ab0 con 0x7f9efc0b1f60 2024-03-29T11:20:47.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:47.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:47.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 10 monclient(hunting): dump: 2024-03-29T11:20:47.325 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:47.325 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:47.325 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:47.325 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:47.325 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:47.326 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:47.326 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:47.326 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:47.326 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:47.326 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:47.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:47.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2072859278 0 0) 0x7f9efc13bb00 con 0x7f9efc149460 2024-03-29T11:20:47.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ee8004eb0 con 0x7f9efc149460 2024-03-29T11:20:47.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556007934 0 0) 0x7f9efc14b0d0 con 0x7f9efc143d10 2024-03-29T11:20:47.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:47.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:47.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:47.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:20:47.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.336+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9efc13bb00 con 0x7f9efc143d10 2024-03-29T11:20:47.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9ef4001e80 con 0x7f9efc149460 2024-03-29T11:20:47.327 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:47.327 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:47.327 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 10 monclient(hunting): dump: 2024-03-29T11:20:47.327 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:47.327 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:47.327 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:47.327 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:47.327 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:47.327 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:47.327 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:47.327 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:47.327 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:47.327 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:47.327 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2745109067 0 0) 0x7f9efc14b300 con 0x7f9efc0b1f60 2024-03-29T11:20:47.327 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 >> v1:172.21.15.170:6790/0 conn(0x7f9efc149460 legacy=0x7f9efc13b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:47.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 >> v1:172.21.15.170:6789/0 conn(0x7f9efc143d10 legacy=0x7f9efc13ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:47.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:47.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 1 monclient: found mon.b 2024-03-29T11:20:47.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:47.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:20:47.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc14b5e0 con 0x7f9efc0b1f60 2024-03-29T11:20:47.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 10 monclient: _finish_auth 0 2024-03-29T11:20:47.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:47.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:47.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9f020f1640 5 monclient: authenticate success, global_id 4519 2024-03-29T11:20:47.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9f020f1640 10 monclient: _renew_subs 2024-03-29T11:20:47.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9f020f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:20:47.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9f020f1640 1 -- 172.21.15.170:0/2736435923 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9efc149b50 con 0x7f9efc0b1f60 2024-03-29T11:20:47.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9f020f1640 10 client.4519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:47.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9f020f1640 10 monclient: _renew_subs 2024-03-29T11:20:47.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9f020f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:20:47.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9f020f1640 1 -- 172.21.15.170:0/2736435923 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9efc14a070 con 0x7f9efc0b1f60 2024-03-29T11:20:47.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9f020f1640 1 librados: init done 2024-03-29T11:20:47.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9eec001c10 con 0x7f9efc0b1f60 2024-03-29T11:20:47.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:47.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f9eec003a60 con 0x7f9efc0b1f60 2024-03-29T11:20:47.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:47.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:47.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ee3fff640 10 monclient: dump: 2024-03-29T11:20:47.329 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:47.329 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:47.329 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:47.329 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:47.329 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:47.329 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:47.329 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:47.329 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:47.330 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:47.330 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:47.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.337+0000 7f9ef9ffb640 4 set_mon_vals no callback set 2024-03-29T11:20:47.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.338+0000 7f9f020f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:47.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.338+0000 7f9f020f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:47.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.338+0000 7f9f020f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:20:47.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.338+0000 7f9f020f1640 1 -- 172.21.15.170:0/2736435923 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9efc1499b0 con 0x7f9efc0b1f60 2024-03-29T11:20:47.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.339+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f9eec003c40 con 0x7f9efc0b1f60 2024-03-29T11:20:47.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.340+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f9eec084bb0 con 0x7f9efc0b1f60 2024-03-29T11:20:47.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.340+0000 7f9ee3fff640 10 client.4519.objecter ms_dispatch 0x7f9efc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:47.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.340+0000 7f9ee3fff640 3 client.4519.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:47.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.340+0000 7f9ee3fff640 3 client.4519.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:47.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.340+0000 7f9ee3fff640 20 client.4519.objecter dump_active .. 0 homeless 2024-03-29T11:20:47.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.340+0000 7f9efaffd640 10 monclient: get_auth_request con 0x7f9ee806df90 auth_method 0 2024-03-29T11:20:47.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.340+0000 7f9ee3fff640 10 client.4519.objecter ms_handle_connect 0x7f9ee806df90 2024-03-29T11:20:47.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.348+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9efc1499b0 con 0x7f9efc0b1f60 2024-03-29T11:20:47.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.348+0000 7f9ee3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:47.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.348+0000 7f9ee3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:47.701 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.725+0000 7f9f020f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:47.702 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.725+0000 7f9f020f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:47.702 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.725+0000 7f9f020f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:20:47.702 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.725+0000 7f9f020f1640 1 -- 172.21.15.170:0/2736435923 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9efc1499b0 con 0x7f9efc0b1f60 2024-03-29T11:20:47.703 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.727+0000 7f9ee3fff640 1 -- 172.21.15.170:0/2736435923 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f9efc1499b0 con 0x7f9efc0b1f60 2024-03-29T11:20:47.703 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.727+0000 7f9ee3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:47.703 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.727+0000 7f9ee3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:47.704 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:20:47.704 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:47.709 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.731+0000 7f9f020f1640 10 librados: watch_flush enter 2024-03-29T11:20:47.709 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.731+0000 7f9f020f1640 10 librados: watch_flush exit 2024-03-29T11:20:47.709 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.731+0000 7f9f020f1640 20 client.4519.objecter shutdown clearing up homeless session... 2024-03-29T11:20:47.709 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.731+0000 7f9f020f1640 10 client.4519.objecter successfully canceled tick 2024-03-29T11:20:47.709 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.731+0000 7f9f020f1640 1 -- 172.21.15.170:0/2736435923 >> v1:172.21.15.170:6816/3298994444 conn(0x7f9ee806df90 legacy=0x7f9ee8070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:47.709 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.731+0000 7f9f020f1640 10 monclient: shutdown 2024-03-29T11:20:47.709 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.731+0000 7f9f020f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:47.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.731+0000 7f9f020f1640 1 -- 172.21.15.170:0/2736435923 >> v1:172.21.15.184:6789/0 conn(0x7f9efc0b1f60 legacy=0x7f9efc147790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:47.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.732+0000 7f9f020f1640 1 -- 172.21.15.170:0/2736435923 shutdown_connections 2024-03-29T11:20:47.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.732+0000 7f9f020f1640 1 -- 172.21.15.170:0/2736435923 >> 172.21.15.170:0/2736435923 conn(0x7f9efc066a00 msgr2=0x7f9efc0b6fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:47.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.732+0000 7f9f020f1640 1 -- 172.21.15.170:0/2736435923 shutdown_connections 2024-03-29T11:20:47.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.732+0000 7f9f020f1640 1 -- 172.21.15.170:0/2736435923 wait complete. 2024-03-29T11:20:47.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:47.732+0000 7f9f020f1640 1 librados: shutdown 2024-03-29T11:20:47.724 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:54.532 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:54.665 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.688+0000 7f0754d2b640 10 monclient: get_monmap_and_config 2024-03-29T11:20:54.665 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.688+0000 7f0754d2b640 10 monclient: build_initial_monmap 2024-03-29T11:20:54.665 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.688+0000 7f0754d2b640 10 monclient: monmap: 2024-03-29T11:20:54.665 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:20:54.665 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:54.665 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:20:54.689712+0000 2024-03-29T11:20:54.665 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:20:54.689712+0000 2024-03-29T11:20:54.666 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:54.666 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:54.666 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:20:54.666 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:20:54.666 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:20:54.666 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:54.667 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.689+0000 7f0754d2b640 1 Processor -- start 2024-03-29T11:20:54.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.689+0000 7f0754d2b640 1 -- start start 2024-03-29T11:20:54.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.689+0000 7f0754d2b640 10 monclient: init 2024-03-29T11:20:54.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.690+0000 7f0754d2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:54.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.690+0000 7f0754d2b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:54.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.690+0000 7f0754d2b640 10 monclient(hunting): picked mon.noname-c con 0x7f0750070120 addr v1:172.21.15.184:6789/0 2024-03-29T11:20:54.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.690+0000 7f0754d2b640 10 monclient(hunting): picked mon.noname-b con 0x7f07500b0420 addr v1:172.21.15.170:6790/0 2024-03-29T11:20:54.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.690+0000 7f0754d2b640 10 monclient(hunting): picked mon.noname-a con 0x7f0750070500 addr v1:172.21.15.170:6789/0 2024-03-29T11:20:54.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.690+0000 7f0754d2b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0750072090 con 0x7f0750070500 2024-03-29T11:20:54.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.690+0000 7f0754d2b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0750072370 con 0x7f0750070120 2024-03-29T11:20:54.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.690+0000 7f0754d2b640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0750072650 con 0x7f07500b0420 2024-03-29T11:20:54.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.690+0000 7f0754d2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:54.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.690+0000 7f0754d2b640 10 monclient(hunting): authenticate will time out at 1436.002304s 2024-03-29T11:20:54.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.690+0000 7f074dd9b640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f07500b0420 0x7f07501405b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:41548/0 (socket says 172.21.15.170:41548) 2024-03-29T11:20:54.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.690+0000 7f074dd9b640 1 -- 172.21.15.170:0/3251006726 learned_addr learned my addr 172.21.15.170:0/3251006726 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:20:54.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.690+0000 7f074d59a640 1 -- 172.21.15.170:0/3251006726 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446232336 0 0) 0x7f0750072650 con 0x7f07500b0420 2024-03-29T11:20:54.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.690+0000 7f074d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:54.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.690+0000 7f074d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:54.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.690+0000 7f074d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:54.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.690+0000 7f074d59a640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:20:54.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.690+0000 7f074d59a640 1 -- 172.21.15.170:0/3251006726 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0738003520 con 0x7f07500b0420 2024-03-29T11:20:54.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.691+0000 7f074d59a640 1 -- 172.21.15.170:0/3251006726 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 22419964 0 0) 0x7f0738003520 con 0x7f07500b0420 2024-03-29T11:20:54.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.691+0000 7f074d59a640 1 -- 172.21.15.170:0/3251006726 >> v1:172.21.15.184:6789/0 conn(0x7f0750070120 legacy=0x7f075013d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:54.669 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.691+0000 7f074d59a640 1 -- 172.21.15.170:0/3251006726 >> v1:172.21.15.170:6789/0 conn(0x7f0750070500 legacy=0x7f075014b810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:54.671 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.692+0000 7f074d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:54.671 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.692+0000 7f074d59a640 1 monclient: found mon.noname-b 2024-03-29T11:20:54.671 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.692+0000 7f074d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:54.671 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.692+0000 7f074d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:20:54.671 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.692+0000 7f074d59a640 1 -- 172.21.15.170:0/3251006726 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0750140090 con 0x7f07500b0420 2024-03-29T11:20:54.671 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.692+0000 7f074d59a640 10 monclient: _finish_auth 0 2024-03-29T11:20:54.671 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.692+0000 7f074d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:54.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.692+0000 7f074d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:54.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.692+0000 7f074d59a640 1 -- 172.21.15.170:0/3251006726 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f0740001690 con 0x7f07500b0420 2024-03-29T11:20:54.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.692+0000 7f074d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:54.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.692+0000 7f074d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:54.672 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.692+0000 7f074d59a640 10 monclient: dump: 2024-03-29T11:20:54.672 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:54.672 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:54.672 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:54.672 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:54.672 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:54.672 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:54.672 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:54.672 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:54.672 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:54.672 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:54.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.692+0000 7f0754d2b640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:20:54.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.692+0000 7f0754d2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:54.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.693+0000 7f074d59a640 1 -- 172.21.15.170:0/3251006726 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07400017f0 con 0x7f07500b0420 2024-03-29T11:20:54.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.693+0000 7f074d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:54.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.693+0000 7f074d59a640 1 -- 172.21.15.170:0/3251006726 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f0740001c40 con 0x7f07500b0420 2024-03-29T11:20:54.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.693+0000 7f074d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:54.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.693+0000 7f074d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:54.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.693+0000 7f074d59a640 10 monclient: dump: 2024-03-29T11:20:54.673 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:54.673 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:54.673 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:54.673 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:54.673 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:54.673 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:54.674 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:54.674 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:54.674 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:54.674 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:54.674 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.693+0000 7f0754d2b640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:54.674 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.693+0000 7f0754d2b640 4 set_mon_vals no callback set 2024-03-29T11:20:54.674 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.694+0000 7f0754d2b640 10 monclient: shutdown 2024-03-29T11:20:54.674 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.694+0000 7f0754d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:54.674 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.694+0000 7f0754d2b640 1 -- 172.21.15.170:0/3251006726 >> v1:172.21.15.170:6790/0 conn(0x7f07500b0420 legacy=0x7f07501405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:54.674 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.694+0000 7f0754d2b640 1 -- 172.21.15.170:0/3251006726 shutdown_connections 2024-03-29T11:20:54.674 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.694+0000 7f0754d2b640 1 -- 172.21.15.170:0/3251006726 >> 172.21.15.170:0/3251006726 conn(0x7f0750067500 msgr2=0x7f07500b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:54.674 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.694+0000 7f0754d2b640 1 -- 172.21.15.170:0/3251006726 shutdown_connections 2024-03-29T11:20:54.674 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.694+0000 7f0754d2b640 1 -- 172.21.15.170:0/3251006726 wait complete. 2024-03-29T11:20:54.679 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.695+0000 7f0754d2b640 10 monclient: build_initial_monmap 2024-03-29T11:20:54.679 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.695+0000 7f0754d2b640 10 monclient: monmap: 2024-03-29T11:20:54.679 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:20:54.679 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:54.679 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:20:54.679 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:20:54.679 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:54.679 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:54.679 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:20:54.679 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:20:54.679 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:20:54.679 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:54.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.696+0000 7f0754d2b640 1 librados: starting msgr at 2024-03-29T11:20:54.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.696+0000 7f0754d2b640 1 librados: starting objecter 2024-03-29T11:20:54.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.697+0000 7f0754d2b640 1 Processor -- start 2024-03-29T11:20:54.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.697+0000 7f0754d2b640 1 -- start start 2024-03-29T11:20:54.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.697+0000 7f0754d2b640 1 librados: setting wanted keys 2024-03-29T11:20:54.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.697+0000 7f0754d2b640 1 librados: calling monclient init 2024-03-29T11:20:54.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.697+0000 7f0754d2b640 10 monclient: init 2024-03-29T11:20:54.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.697+0000 7f0754d2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:54.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.697+0000 7f0754d2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:54.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.697+0000 7f0754d2b640 10 monclient(hunting): picked mon.noname-b con 0x7f0750070120 addr v1:172.21.15.184:6789/0 2024-03-29T11:20:54.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.697+0000 7f0754d2b640 10 monclient(hunting): picked mon.noname-a con 0x7f0750070500 addr v1:172.21.15.170:6789/0 2024-03-29T11:20:54.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f0754d2b640 10 monclient(hunting): picked mon.noname-c con 0x7f07500b0420 addr v1:172.21.15.170:6790/0 2024-03-29T11:20:54.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f0754d2b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07500adc20 con 0x7f0750070500 2024-03-29T11:20:54.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f0754d2b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07500acb30 con 0x7f0750070120 2024-03-29T11:20:54.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f074ed9d640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f07500b0420 0x7f07501e0080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:42598/0 (socket says 172.21.15.170:42598) 2024-03-29T11:20:54.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f074ed9d640 1 -- 172.21.15.170:0/2986364083 learned_addr learned my addr 172.21.15.170:0/2986364083 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:20:54.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f0754d2b640 1 -- 172.21.15.170:0/2986364083 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07500ace10 con 0x7f07500b0420 2024-03-29T11:20:54.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 10 client.?.objecter ms_handle_connect 0x7f07500b0420 2024-03-29T11:20:54.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:54.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f0754d2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:54.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f0754d2b640 10 monclient(hunting): authenticate will time out at 1436.010312s 2024-03-29T11:20:54.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 1 -- 172.21.15.170:0/2986364083 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469915935 0 0) 0x7f07500ace10 con 0x7f07500b0420 2024-03-29T11:20:54.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:54.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:54.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:54.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:20:54.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 1 -- 172.21.15.170:0/2986364083 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0730004eb0 con 0x7f07500b0420 2024-03-29T11:20:54.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 10 client.?.objecter ms_handle_connect 0x7f0750070120 2024-03-29T11:20:54.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:54.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 1 -- 172.21.15.170:0/2986364083 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946662804 0 0) 0x7f07500acb30 con 0x7f0750070120 2024-03-29T11:20:54.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:54.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:54.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:54.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:20:54.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 1 -- 172.21.15.170:0/2986364083 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07500ace10 con 0x7f0750070120 2024-03-29T11:20:54.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 1 -- 172.21.15.170:0/2986364083 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2793985920 0 0) 0x7f0730004eb0 con 0x7f07500b0420 2024-03-29T11:20:54.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 1 -- 172.21.15.170:0/2986364083 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07500acb30 con 0x7f07500b0420 2024-03-29T11:20:54.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 1 -- 172.21.15.170:0/2986364083 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f0748004380 con 0x7f07500b0420 2024-03-29T11:20:54.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:54.683 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:54.683 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 10 monclient(hunting): dump: 2024-03-29T11:20:54.683 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:54.683 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:54.683 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:54.683 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:54.683 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:54.683 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:54.683 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:54.683 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:54.683 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:54.683 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:54.683 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.698+0000 7f073effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:54.683 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.699+0000 7f073effd640 1 -- 172.21.15.170:0/2986364083 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3142694112 0 0) 0x7f07500ace10 con 0x7f0750070120 2024-03-29T11:20:54.683 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.699+0000 7f073effd640 1 -- 172.21.15.170:0/2986364083 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0730004eb0 con 0x7f0750070120 2024-03-29T11:20:54.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.699+0000 7f073effd640 1 -- 172.21.15.170:0/2986364083 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2452438856 0 0) 0x7f07500acb30 con 0x7f07500b0420 2024-03-29T11:20:54.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.699+0000 7f073effd640 1 -- 172.21.15.170:0/2986364083 >> v1:172.21.15.184:6789/0 conn(0x7f0750070120 legacy=0x7f07500abaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:54.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.699+0000 7f073effd640 1 -- 172.21.15.170:0/2986364083 >> v1:172.21.15.170:6789/0 conn(0x7f0750070500 legacy=0x7f07500ad510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:54.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.700+0000 7f073effd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:54.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.700+0000 7f073effd640 1 monclient: found mon.c 2024-03-29T11:20:54.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.700+0000 7f073effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:54.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.700+0000 7f073effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:20:54.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.700+0000 7f073effd640 1 -- 172.21.15.170:0/2986364083 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07500ad0f0 con 0x7f07500b0420 2024-03-29T11:20:54.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.700+0000 7f073effd640 10 monclient: _finish_auth 0 2024-03-29T11:20:54.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.700+0000 7f073effd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:54.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.700+0000 7f073effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:54.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.700+0000 7f0754d2b640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:20:54.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.700+0000 7f0754d2b640 10 monclient: _renew_subs 2024-03-29T11:20:54.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.700+0000 7f0754d2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:20:54.685 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.700+0000 7f0754d2b640 1 -- 172.21.15.170:0/2986364083 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07500ab5d0 con 0x7f07500b0420 2024-03-29T11:20:54.685 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.700+0000 7f0754d2b640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:54.685 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.700+0000 7f0754d2b640 10 monclient: _renew_subs 2024-03-29T11:20:54.685 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.700+0000 7f0754d2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:20:54.685 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.700+0000 7f0754d2b640 1 -- 172.21.15.170:0/2986364083 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07500aacd0 con 0x7f07500b0420 2024-03-29T11:20:54.685 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.700+0000 7f0754d2b640 1 librados: init done 2024-03-29T11:20:54.685 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.701+0000 7f073effd640 1 -- 172.21.15.170:0/2986364083 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0748001250 con 0x7f07500b0420 2024-03-29T11:20:54.685 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.701+0000 7f073effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:54.685 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.701+0000 7f073effd640 1 -- 172.21.15.170:0/2986364083 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f0748001620 con 0x7f07500b0420 2024-03-29T11:20:54.685 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.701+0000 7f073effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:54.685 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.701+0000 7f073effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:54.685 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.701+0000 7f073effd640 10 monclient: dump: 2024-03-29T11:20:54.685 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:20:54.685 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:20:54.685 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:54.686 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:20:54.686 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:54.686 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:20:54.686 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:20:54.686 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:20:54.686 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:20:54.686 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:20:54.686 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.701+0000 7f074cd99640 4 set_mon_vals no callback set 2024-03-29T11:20:54.686 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.701+0000 7f0754d2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:54.686 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.702+0000 7f0754d2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:54.686 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.702+0000 7f0754d2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:20:54.686 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.702+0000 7f0754d2b640 1 -- 172.21.15.170:0/2986364083 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0734005080 con 0x7f07500b0420 2024-03-29T11:20:54.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.702+0000 7f073effd640 1 -- 172.21.15.170:0/2986364083 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f0748001780 con 0x7f07500b0420 2024-03-29T11:20:54.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.703+0000 7f073effd640 1 -- 172.21.15.170:0/2986364083 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f0748084d70 con 0x7f07500b0420 2024-03-29T11:20:54.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.703+0000 7f073effd640 10 client.4601.objecter ms_dispatch 0x7f0750000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:54.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.703+0000 7f073effd640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:54.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.703+0000 7f073effd640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:54.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.703+0000 7f073effd640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-29T11:20:54.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.704+0000 7f074e59c640 10 monclient: get_auth_request con 0x7f073006dbb0 auth_method 0 2024-03-29T11:20:54.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.704+0000 7f073effd640 10 client.4601.objecter ms_handle_connect 0x7f073006dbb0 2024-03-29T11:20:54.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.710+0000 7f073effd640 1 -- 172.21.15.170:0/2986364083 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0748050650 con 0x7f07500b0420 2024-03-29T11:20:54.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.710+0000 7f073effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:54.689 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:54.710+0000 7f073effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:54.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.015+0000 7f0754d2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:54.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.015+0000 7f0754d2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:54.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.015+0000 7f0754d2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:20:54.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.015+0000 7f0754d2b640 1 -- 172.21.15.170:0/2986364083 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0734005330 con 0x7f07500b0420 2024-03-29T11:20:54.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.016+0000 7f073effd640 1 -- 172.21.15.170:0/2986364083 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f0748052580 con 0x7f07500b0420 2024-03-29T11:20:54.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.016+0000 7f073effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:54.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.016+0000 7f073effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:54.993 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:20:54.994 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:54.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.019+0000 7f0754d2b640 10 librados: watch_flush enter 2024-03-29T11:20:54.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.020+0000 7f0754d2b640 10 librados: watch_flush exit 2024-03-29T11:20:54.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.020+0000 7f0754d2b640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-29T11:20:54.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.020+0000 7f0754d2b640 10 client.4601.objecter successfully canceled tick 2024-03-29T11:20:54.998 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.020+0000 7f0754d2b640 1 -- 172.21.15.170:0/2986364083 >> v1:172.21.15.170:6816/3298994444 conn(0x7f073006dbb0 legacy=0x7f0730070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:54.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.020+0000 7f0754d2b640 10 monclient: shutdown 2024-03-29T11:20:54.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.021+0000 7f0754d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:54.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.021+0000 7f0754d2b640 1 -- 172.21.15.170:0/2986364083 >> v1:172.21.15.170:6790/0 conn(0x7f07500b0420 legacy=0x7f07501e0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:54.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.021+0000 7f0754d2b640 1 -- 172.21.15.170:0/2986364083 shutdown_connections 2024-03-29T11:20:54.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.021+0000 7f0754d2b640 1 -- 172.21.15.170:0/2986364083 >> 172.21.15.170:0/2986364083 conn(0x7f0750067500 msgr2=0x7f075013c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:54.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.021+0000 7f0754d2b640 1 -- 172.21.15.170:0/2986364083 shutdown_connections 2024-03-29T11:20:54.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.021+0000 7f0754d2b640 1 -- 172.21.15.170:0/2986364083 wait complete. 2024-03-29T11:20:54.999 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:20:55.021+0000 7f0754d2b640 1 librados: shutdown 2024-03-29T11:20:55.014 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:01.821 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:01.957 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.980+0000 7fe0b340f640 10 monclient: get_monmap_and_config 2024-03-29T11:21:01.958 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.980+0000 7fe0b340f640 10 monclient: build_initial_monmap 2024-03-29T11:21:01.958 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.980+0000 7fe0b340f640 10 monclient: monmap: 2024-03-29T11:21:01.958 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:21:01.958 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:01.958 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:21:01.982256+0000 2024-03-29T11:21:01.958 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:21:01.982256+0000 2024-03-29T11:21:01.958 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:01.958 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:01.958 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:21:01.958 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:21:01.958 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:21:01.959 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:01.959 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.981+0000 7fe0b340f640 1 Processor -- start 2024-03-29T11:21:01.959 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.982+0000 7fe0b340f640 1 -- start start 2024-03-29T11:21:01.959 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.982+0000 7fe0b340f640 10 monclient: init 2024-03-29T11:21:01.961 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.983+0000 7fe0b340f640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:01.961 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.983+0000 7fe0b340f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:01.961 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.983+0000 7fe0b340f640 10 monclient(hunting): picked mon.noname-c con 0x7fe0ac1457f0 addr v1:172.21.15.184:6789/0 2024-03-29T11:21:01.961 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.983+0000 7fe0b340f640 10 monclient(hunting): picked mon.noname-b con 0x7fe0ac13f650 addr v1:172.21.15.170:6790/0 2024-03-29T11:21:01.961 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.983+0000 7fe0b340f640 10 monclient(hunting): picked mon.noname-a con 0x7fe0ac0a9d70 addr v1:172.21.15.170:6789/0 2024-03-29T11:21:01.961 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.983+0000 7fe0b340f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac0aaef0 con 0x7fe0ac0a9d70 2024-03-29T11:21:01.961 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.983+0000 7fe0b340f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac145bd0 con 0x7fe0ac1457f0 2024-03-29T11:21:01.961 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.983+0000 7fe0b340f640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac145eb0 con 0x7fe0ac13f650 2024-03-29T11:21:01.961 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.983+0000 7fe0b340f640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:01.961 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.983+0000 7fe0b340f640 10 monclient(hunting): authenticate will time out at 1443.295360s 2024-03-29T11:21:01.961 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.983+0000 7fe0b09aa640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fe0ac13f650 0x7fe0ac0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:49068/0 (socket says 172.21.15.170:49068) 2024-03-29T11:21:01.961 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.983+0000 7fe0b09aa640 1 -- 172.21.15.170:0/1809929461 learned_addr learned my addr 172.21.15.170:0/1809929461 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:21:01.961 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.983+0000 7fe0a3fff640 1 -- 172.21.15.170:0/1809929461 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393609696 0 0) 0x7fe0ac145eb0 con 0x7fe0ac13f650 2024-03-29T11:21:01.962 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.983+0000 7fe0a3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:01.962 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.983+0000 7fe0a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:01.962 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.983+0000 7fe0a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:01.962 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.983+0000 7fe0a3fff640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:21:01.962 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.983+0000 7fe0a3fff640 1 -- 172.21.15.170:0/1809929461 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe09800bd30 con 0x7fe0ac13f650 2024-03-29T11:21:01.962 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.984+0000 7fe0a3fff640 1 -- 172.21.15.170:0/1809929461 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1565156684 0 0) 0x7fe09800bd30 con 0x7fe0ac13f650 2024-03-29T11:21:01.962 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.984+0000 7fe0a3fff640 1 -- 172.21.15.170:0/1809929461 >> v1:172.21.15.184:6789/0 conn(0x7fe0ac1457f0 legacy=0x7fe0ac13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:01.962 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.984+0000 7fe0a3fff640 1 -- 172.21.15.170:0/1809929461 >> v1:172.21.15.170:6789/0 conn(0x7fe0ac0a9d70 legacy=0x7fe0ac14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:01.965 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.985+0000 7fe0a3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:01.965 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.985+0000 7fe0a3fff640 1 monclient: found mon.noname-b 2024-03-29T11:21:01.965 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.985+0000 7fe0a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:01.965 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.985+0000 7fe0a3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:21:01.965 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.985+0000 7fe0a3fff640 1 -- 172.21.15.170:0/1809929461 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac146340 con 0x7fe0ac13f650 2024-03-29T11:21:01.965 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.985+0000 7fe0a3fff640 10 monclient: _finish_auth 0 2024-03-29T11:21:01.965 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.985+0000 7fe0a3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:01.965 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.985+0000 7fe0a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:01.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.985+0000 7fe0a3fff640 1 -- 172.21.15.170:0/1809929461 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe09c001690 con 0x7fe0ac13f650 2024-03-29T11:21:01.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.985+0000 7fe0a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:01.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.985+0000 7fe0a3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:01.966 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.985+0000 7fe0a3fff640 10 monclient: dump: 2024-03-29T11:21:01.966 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:01.966 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:01.966 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:01.966 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:01.966 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:01.966 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:01.966 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:01.966 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:01.966 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:01.966 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:01.967 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.985+0000 7fe0a3fff640 1 -- 172.21.15.170:0/1809929461 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe09c001920 con 0x7fe0ac13f650 2024-03-29T11:21:01.967 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.985+0000 7fe0a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:01.967 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.985+0000 7fe0a3fff640 1 -- 172.21.15.170:0/1809929461 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe09c001cb0 con 0x7fe0ac13f650 2024-03-29T11:21:01.967 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.985+0000 7fe0a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:01.967 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.985+0000 7fe0a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:01.967 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.985+0000 7fe0a3fff640 10 monclient: dump: 2024-03-29T11:21:01.967 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:01.967 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:01.967 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:01.967 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:01.967 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:01.967 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:01.967 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:01.967 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:01.967 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:01.968 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:01.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.986+0000 7fe0b340f640 5 monclient: authenticate success, global_id 4604 2024-03-29T11:21:01.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.986+0000 7fe0b340f640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:01.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.986+0000 7fe0b340f640 4 set_mon_vals no callback set 2024-03-29T11:21:01.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.987+0000 7fe0b340f640 10 monclient: shutdown 2024-03-29T11:21:01.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.987+0000 7fe0b340f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:01.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.987+0000 7fe0b340f640 1 -- 172.21.15.170:0/1809929461 >> v1:172.21.15.170:6790/0 conn(0x7fe0ac13f650 legacy=0x7fe0ac0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:01.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.987+0000 7fe0b340f640 1 -- 172.21.15.170:0/1809929461 shutdown_connections 2024-03-29T11:21:01.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.987+0000 7fe0b340f640 1 -- 172.21.15.170:0/1809929461 >> 172.21.15.170:0/1809929461 conn(0x7fe0ac066a00 msgr2=0x7fe0ac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:01.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.988+0000 7fe0b340f640 1 -- 172.21.15.170:0/1809929461 shutdown_connections 2024-03-29T11:21:01.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.988+0000 7fe0b340f640 1 -- 172.21.15.170:0/1809929461 wait complete. 2024-03-29T11:21:01.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.988+0000 7fe0b340f640 10 monclient: build_initial_monmap 2024-03-29T11:21:01.968 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.988+0000 7fe0b340f640 10 monclient: monmap: 2024-03-29T11:21:01.968 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:21:01.969 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:01.969 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:21:01.969 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:21:01.969 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:01.969 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:01.969 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:21:01.969 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:21:01.969 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:21:01.969 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:01.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.988+0000 7fe0b340f640 1 librados: starting msgr at 2024-03-29T11:21:01.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.989+0000 7fe0b340f640 1 librados: starting objecter 2024-03-29T11:21:01.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.989+0000 7fe0b340f640 1 Processor -- start 2024-03-29T11:21:01.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.989+0000 7fe0b340f640 1 -- start start 2024-03-29T11:21:01.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.989+0000 7fe0b340f640 1 librados: setting wanted keys 2024-03-29T11:21:01.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.989+0000 7fe0b340f640 1 librados: calling monclient init 2024-03-29T11:21:01.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.989+0000 7fe0b340f640 10 monclient: init 2024-03-29T11:21:01.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.989+0000 7fe0b340f640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:01.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.989+0000 7fe0b340f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:01.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.990+0000 7fe0b340f640 10 monclient(hunting): picked mon.noname-b con 0x7fe0ac0a9d70 addr v1:172.21.15.184:6789/0 2024-03-29T11:21:01.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.990+0000 7fe0b340f640 10 monclient(hunting): picked mon.noname-c con 0x7fe0ac13f650 addr v1:172.21.15.170:6790/0 2024-03-29T11:21:01.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.990+0000 7fe0b11ab640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fe0ac0a9d70 0x7fe0ac143190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:60446/0 (socket says 172.21.15.170:60446) 2024-03-29T11:21:01.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.990+0000 7fe0b09aa640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fe0ac13f650 0x7fe0ac12f900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:37656/0 (socket says 172.21.15.170:37656) 2024-03-29T11:21:01.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.990+0000 7fe0b09aa640 1 -- 172.21.15.170:0/2614624230 learned_addr learned my addr 172.21.15.170:0/2614624230 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:21:01.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.990+0000 7fe0a1ffb640 10 client.?.objecter ms_handle_connect 0x7fe0ac13f650 2024-03-29T11:21:01.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.990+0000 7fe0a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:01.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.990+0000 7fe0b340f640 10 monclient(hunting): picked mon.noname-a con 0x7fe0ac1457f0 addr v1:172.21.15.170:6789/0 2024-03-29T11:21:01.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.990+0000 7fe0a1ffb640 10 client.?.objecter ms_handle_connect 0x7fe0ac0a9d70 2024-03-29T11:21:01.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.990+0000 7fe0a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:01.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.990+0000 7fe0b340f640 1 -- 172.21.15.170:0/2614624230 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac13d2a0 con 0x7fe0ac1457f0 2024-03-29T11:21:01.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.991+0000 7fe0b340f640 1 -- 172.21.15.170:0/2614624230 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac130720 con 0x7fe0ac0a9d70 2024-03-29T11:21:01.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.991+0000 7fe0b340f640 1 -- 172.21.15.170:0/2614624230 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac144ed0 con 0x7fe0ac13f650 2024-03-29T11:21:01.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.991+0000 7fe0b340f640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:01.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.991+0000 7fe0b340f640 10 monclient(hunting): authenticate will time out at 1443.303332s 2024-03-29T11:21:01.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.991+0000 7fe0a1ffb640 1 -- 172.21.15.170:0/2614624230 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133934423 0 0) 0x7fe0ac144ed0 con 0x7fe0ac13f650 2024-03-29T11:21:01.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.991+0000 7fe0a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:01.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.991+0000 7fe0a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:01.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.991+0000 7fe0a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:01.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.991+0000 7fe0a1ffb640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:21:01.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.991+0000 7fe0a1ffb640 1 -- 172.21.15.170:0/2614624230 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe090004eb0 con 0x7fe0ac13f650 2024-03-29T11:21:01.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.991+0000 7fe0a1ffb640 1 -- 172.21.15.170:0/2614624230 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 89985461 0 0) 0x7fe0ac130720 con 0x7fe0ac0a9d70 2024-03-29T11:21:01.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.991+0000 7fe0a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:01.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.991+0000 7fe0a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:01.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.991+0000 7fe0a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:01.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.991+0000 7fe0a1ffb640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:21:01.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.991+0000 7fe0a1ffb640 1 -- 172.21.15.170:0/2614624230 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0ac144ed0 con 0x7fe0ac0a9d70 2024-03-29T11:21:01.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.992+0000 7fe0a1ffb640 1 -- 172.21.15.170:0/2614624230 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2611471218 0 0) 0x7fe090004eb0 con 0x7fe0ac13f650 2024-03-29T11:21:01.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.992+0000 7fe0a1ffb640 1 -- 172.21.15.170:0/2614624230 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe090005490 con 0x7fe0ac13f650 2024-03-29T11:21:01.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.992+0000 7fe0a1ffb640 1 -- 172.21.15.170:0/2614624230 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe09c001610 con 0x7fe0ac13f650 2024-03-29T11:21:01.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.992+0000 7fe0a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:01.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.992+0000 7fe0a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:01.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.992+0000 7fe0a1ffb640 10 monclient(hunting): dump: 2024-03-29T11:21:01.978 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:01.979 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:01.979 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:01.979 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:01.979 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:01.979 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:01.979 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:01.979 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:01.979 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:01.979 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:01.979 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.992+0000 7fe0a1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:01.979 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.992+0000 7fe0a1ffb640 1 -- 172.21.15.170:0/2614624230 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1336607116 0 0) 0x7fe090005490 con 0x7fe0ac13f650 2024-03-29T11:21:01.979 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.992+0000 7fe0a1ffb640 1 -- 172.21.15.170:0/2614624230 >> v1:172.21.15.184:6789/0 conn(0x7fe0ac0a9d70 legacy=0x7fe0ac143190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:01.979 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.992+0000 7fe0a1ffb640 1 -- 172.21.15.170:0/2614624230 >> v1:172.21.15.170:6789/0 conn(0x7fe0ac1457f0 legacy=0x7fe0ac130010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:01.979 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.993+0000 7fe0a1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:01.979 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.993+0000 7fe0a1ffb640 1 monclient: found mon.c 2024-03-29T11:21:01.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.993+0000 7fe0a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:01.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.993+0000 7fe0a1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:01.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.993+0000 7fe0a1ffb640 1 -- 172.21.15.170:0/2614624230 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac1451b0 con 0x7fe0ac13f650 2024-03-29T11:21:01.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.993+0000 7fe0a1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:01.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.993+0000 7fe0a1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:01.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.993+0000 7fe0a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:01.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.993+0000 7fe0b340f640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:21:01.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.993+0000 7fe0b340f640 10 monclient: _renew_subs 2024-03-29T11:21:01.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.993+0000 7fe0b340f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:01.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.993+0000 7fe0b340f640 1 -- 172.21.15.170:0/2614624230 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ac13ce50 con 0x7fe0ac13f650 2024-03-29T11:21:01.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.993+0000 7fe0b340f640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:01.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.993+0000 7fe0b340f640 10 monclient: _renew_subs 2024-03-29T11:21:01.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.994+0000 7fe0a1ffb640 1 -- 172.21.15.170:0/2614624230 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe09c004bf0 con 0x7fe0ac13f650 2024-03-29T11:21:01.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.993+0000 7fe0b340f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:01.980 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.994+0000 7fe0b340f640 1 -- 172.21.15.170:0/2614624230 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ac13d730 con 0x7fe0ac13f650 2024-03-29T11:21:01.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.994+0000 7fe0a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:01.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.994+0000 7fe0a1ffb640 1 -- 172.21.15.170:0/2614624230 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe09c004f00 con 0x7fe0ac13f650 2024-03-29T11:21:01.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.994+0000 7fe0a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:01.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.994+0000 7fe0a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:01.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.994+0000 7fe0a1ffb640 10 monclient: dump: 2024-03-29T11:21:01.981 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:01.981 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:01.981 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:01.981 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:01.981 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:01.981 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:01.981 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:01.981 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:01.981 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:01.981 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:01.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.994+0000 7fe0b340f640 1 librados: init done 2024-03-29T11:21:01.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.994+0000 7fe0a37fe640 4 set_mon_vals no callback set 2024-03-29T11:21:01.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.995+0000 7fe0b340f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:01.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.995+0000 7fe0b340f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:01.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.995+0000 7fe0b340f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:01.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.995+0000 7fe0b340f640 1 -- 172.21.15.170:0/2614624230 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ac13d040 con 0x7fe0ac13f650 2024-03-29T11:21:01.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.995+0000 7fe0a1ffb640 1 -- 172.21.15.170:0/2614624230 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fe09c01aed0 con 0x7fe0ac13f650 2024-03-29T11:21:01.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.996+0000 7fe0a1ffb640 1 -- 172.21.15.170:0/2614624230 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fe09c084ba0 con 0x7fe0ac13f650 2024-03-29T11:21:01.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.996+0000 7fe0a1ffb640 10 client.4607.objecter ms_dispatch 0x7fe0ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:01.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.996+0000 7fe0a1ffb640 3 client.4607.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:01.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.996+0000 7fe0a1ffb640 3 client.4607.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:01.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.996+0000 7fe0b11ab640 10 monclient: get_auth_request con 0x7fe09006dfc0 auth_method 0 2024-03-29T11:21:01.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.996+0000 7fe0a1ffb640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-29T11:21:01.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:01.996+0000 7fe0a1ffb640 10 client.4607.objecter ms_handle_connect 0x7fe09006dfc0 2024-03-29T11:21:01.983 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.004+0000 7fe0a1ffb640 1 -- 172.21.15.170:0/2614624230 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe09c0504e0 con 0x7fe0ac13f650 2024-03-29T11:21:01.983 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.004+0000 7fe0a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:01.983 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.004+0000 7fe0a1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:02.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.312+0000 7fe0b340f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:02.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.312+0000 7fe0b340f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:02.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.313+0000 7fe0b340f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:02.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.313+0000 7fe0b340f640 1 -- 172.21.15.170:0/2614624230 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0ac075f10 con 0x7fe0ac13f650 2024-03-29T11:21:02.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.314+0000 7fe0a1ffb640 1 -- 172.21.15.170:0/2614624230 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fe09c084250 con 0x7fe0ac13f650 2024-03-29T11:21:02.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.314+0000 7fe0a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:02.291 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.314+0000 7fe0a1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:02.291 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:21:02.291 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:02.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.316+0000 7fe0b340f640 10 librados: watch_flush enter 2024-03-29T11:21:02.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.316+0000 7fe0b340f640 10 librados: watch_flush exit 2024-03-29T11:21:02.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.316+0000 7fe0b340f640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-29T11:21:02.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.316+0000 7fe0b340f640 10 client.4607.objecter successfully canceled tick 2024-03-29T11:21:02.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.316+0000 7fe0b340f640 1 -- 172.21.15.170:0/2614624230 >> v1:172.21.15.170:6816/3298994444 conn(0x7fe09006dfc0 legacy=0x7fe090070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:02.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.316+0000 7fe0b340f640 10 monclient: shutdown 2024-03-29T11:21:02.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.316+0000 7fe0b340f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:02.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.316+0000 7fe0b340f640 1 -- 172.21.15.170:0/2614624230 >> v1:172.21.15.170:6790/0 conn(0x7fe0ac13f650 legacy=0x7fe0ac12f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:02.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.317+0000 7fe0b340f640 1 -- 172.21.15.170:0/2614624230 shutdown_connections 2024-03-29T11:21:02.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.317+0000 7fe0b340f640 1 -- 172.21.15.170:0/2614624230 >> 172.21.15.170:0/2614624230 conn(0x7fe0ac066a00 msgr2=0x7fe0ac1360b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:02.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.317+0000 7fe0b340f640 1 -- 172.21.15.170:0/2614624230 shutdown_connections 2024-03-29T11:21:02.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.317+0000 7fe0b340f640 1 -- 172.21.15.170:0/2614624230 wait complete. 2024-03-29T11:21:02.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:02.317+0000 7fe0b340f640 1 librados: shutdown 2024-03-29T11:21:02.309 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:09.116 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:09.268 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.291+0000 7f169336d640 10 monclient: get_monmap_and_config 2024-03-29T11:21:09.269 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.291+0000 7f169336d640 10 monclient: build_initial_monmap 2024-03-29T11:21:09.269 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.291+0000 7f169336d640 10 monclient: monmap: 2024-03-29T11:21:09.269 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:21:09.269 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:09.269 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:21:09.293032+0000 2024-03-29T11:21:09.269 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:21:09.293032+0000 2024-03-29T11:21:09.269 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:09.269 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:09.269 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:21:09.269 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:21:09.269 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:21:09.269 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:09.270 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.292+0000 7f169336d640 1 Processor -- start 2024-03-29T11:21:09.270 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.293+0000 7f169336d640 1 -- start start 2024-03-29T11:21:09.270 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.293+0000 7f169336d640 10 monclient: init 2024-03-29T11:21:09.278 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.293+0000 7f169336d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:09.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.293+0000 7f169336d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:09.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.296+0000 7f169336d640 10 monclient(hunting): picked mon.noname-c con 0x7f1684006a90 addr v1:172.21.15.184:6789/0 2024-03-29T11:21:09.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.296+0000 7f169336d640 10 monclient(hunting): picked mon.noname-a con 0x7f168402b120 addr v1:172.21.15.170:6789/0 2024-03-29T11:21:09.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.296+0000 7f1691109640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f1684006a90 0x7f16840c8d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:35331/0 (socket says 172.21.15.170:35331) 2024-03-29T11:21:09.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.296+0000 7f1691109640 1 -- 172.21.15.170:0/2418727218 learned_addr learned my addr 172.21.15.170:0/2418727218 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:21:09.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f169336d640 10 monclient(hunting): picked mon.noname-b con 0x7f1684029c50 addr v1:172.21.15.170:6790/0 2024-03-29T11:21:09.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f169336d640 1 -- 172.21.15.170:0/2418727218 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f168402a290 con 0x7f168402b120 2024-03-29T11:21:09.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f169336d640 1 -- 172.21.15.170:0/2418727218 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f168402c460 con 0x7f1684006a90 2024-03-29T11:21:09.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f169336d640 1 -- 172.21.15.170:0/2418727218 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16840c9460 con 0x7f1684029c50 2024-03-29T11:21:09.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f169336d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:09.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f169336d640 10 monclient(hunting): authenticate will time out at 1450.609189s 2024-03-29T11:21:09.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f1683fff640 1 -- 172.21.15.170:0/2418727218 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596701391 0 0) 0x7f16840c9460 con 0x7f1684029c50 2024-03-29T11:21:09.279 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f1683fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:09.280 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f1683fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:09.280 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f1683fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:09.280 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f1683fff640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:21:09.280 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f1683fff640 1 -- 172.21.15.170:0/2418727218 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1678003520 con 0x7f1684029c50 2024-03-29T11:21:09.280 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f1683fff640 1 -- 172.21.15.170:0/2418727218 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775592120 0 0) 0x7f168402c460 con 0x7f1684006a90 2024-03-29T11:21:09.280 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f1683fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:09.280 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f1683fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:09.280 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f1683fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:09.280 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f1683fff640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:21:09.280 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f1683fff640 1 -- 172.21.15.170:0/2418727218 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16840c9460 con 0x7f1684006a90 2024-03-29T11:21:09.280 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f1683fff640 1 -- 172.21.15.170:0/2418727218 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1866262749 0 0) 0x7f1678003520 con 0x7f1684029c50 2024-03-29T11:21:09.280 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f1683fff640 1 -- 172.21.15.170:0/2418727218 >> v1:172.21.15.184:6789/0 conn(0x7f1684006a90 legacy=0x7f16840c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:09.280 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.297+0000 7f1683fff640 1 -- 172.21.15.170:0/2418727218 >> v1:172.21.15.170:6789/0 conn(0x7f168402b120 legacy=0x7f16840cf120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:09.280 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.298+0000 7f1683fff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:09.281 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.298+0000 7f1683fff640 1 monclient: found mon.noname-b 2024-03-29T11:21:09.281 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.298+0000 7f1683fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:09.281 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.298+0000 7f1683fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:21:09.281 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.298+0000 7f1683fff640 1 -- 172.21.15.170:0/2418727218 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16840c95f0 con 0x7f1684029c50 2024-03-29T11:21:09.281 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.299+0000 7f1683fff640 10 monclient: _finish_auth 0 2024-03-29T11:21:09.281 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.299+0000 7f1683fff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:09.281 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.299+0000 7f1683fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:09.281 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.299+0000 7f1683fff640 1 -- 172.21.15.170:0/2418727218 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f168c04f730 con 0x7f1684029c50 2024-03-29T11:21:09.281 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.299+0000 7f1683fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:09.281 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.299+0000 7f1683fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:09.281 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.299+0000 7f1683fff640 10 monclient: dump: 2024-03-29T11:21:09.281 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:09.281 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:09.281 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:09.281 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:09.282 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:09.282 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:09.282 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:09.282 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:09.282 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:09.282 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:09.282 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.299+0000 7f1683fff640 1 -- 172.21.15.170:0/2418727218 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f168c0008d0 con 0x7f1684029c50 2024-03-29T11:21:09.282 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.299+0000 7f1683fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:09.282 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.299+0000 7f1683fff640 1 -- 172.21.15.170:0/2418727218 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f168c0758b0 con 0x7f1684029c50 2024-03-29T11:21:09.282 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.299+0000 7f1683fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:09.282 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.299+0000 7f1683fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:09.282 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.299+0000 7f1683fff640 10 monclient: dump: 2024-03-29T11:21:09.282 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:09.283 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:09.283 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:09.283 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:09.283 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:09.283 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:09.283 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:09.283 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:09.283 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:09.283 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:09.283 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.300+0000 7f169336d640 5 monclient: authenticate success, global_id 4610 2024-03-29T11:21:09.283 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.300+0000 7f169336d640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:09.283 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.300+0000 7f169336d640 4 set_mon_vals no callback set 2024-03-29T11:21:09.283 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.301+0000 7f169336d640 10 monclient: shutdown 2024-03-29T11:21:09.284 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.301+0000 7f169336d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:09.284 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.301+0000 7f169336d640 1 -- 172.21.15.170:0/2418727218 >> v1:172.21.15.170:6790/0 conn(0x7f1684029c50 legacy=0x7f16840d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:09.284 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.301+0000 7f169336d640 1 -- 172.21.15.170:0/2418727218 shutdown_connections 2024-03-29T11:21:09.284 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.301+0000 7f169336d640 1 -- 172.21.15.170:0/2418727218 >> 172.21.15.170:0/2418727218 conn(0x7f1684005910 msgr2=0x7f1684005d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:09.284 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.301+0000 7f169336d640 1 -- 172.21.15.170:0/2418727218 shutdown_connections 2024-03-29T11:21:09.286 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.301+0000 7f169336d640 1 -- 172.21.15.170:0/2418727218 wait complete. 2024-03-29T11:21:09.286 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.302+0000 7f169336d640 10 monclient: build_initial_monmap 2024-03-29T11:21:09.287 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.302+0000 7f169336d640 10 monclient: monmap: 2024-03-29T11:21:09.287 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:21:09.287 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:09.287 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:21:09.287 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:21:09.287 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:09.287 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:09.287 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:21:09.287 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:21:09.287 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:21:09.287 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:09.287 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.303+0000 7f169336d640 1 librados: starting msgr at 2024-03-29T11:21:09.287 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.303+0000 7f169336d640 1 librados: starting objecter 2024-03-29T11:21:09.287 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.303+0000 7f169336d640 1 Processor -- start 2024-03-29T11:21:09.287 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.303+0000 7f169336d640 1 -- start start 2024-03-29T11:21:09.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.303+0000 7f169336d640 1 librados: setting wanted keys 2024-03-29T11:21:09.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.303+0000 7f169336d640 1 librados: calling monclient init 2024-03-29T11:21:09.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.303+0000 7f169336d640 10 monclient: init 2024-03-29T11:21:09.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.304+0000 7f169336d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:09.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.304+0000 7f169336d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:09.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.305+0000 7f169336d640 10 monclient(hunting): picked mon.noname-a con 0x7f1684006a90 addr v1:172.21.15.170:6789/0 2024-03-29T11:21:09.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.305+0000 7f169336d640 10 monclient(hunting): picked mon.noname-c con 0x7f1684029c50 addr v1:172.21.15.170:6790/0 2024-03-29T11:21:09.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.305+0000 7f169336d640 10 monclient(hunting): picked mon.noname-b con 0x7f16840bff20 addr v1:172.21.15.184:6789/0 2024-03-29T11:21:09.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.305+0000 7f169336d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16840cee70 con 0x7f1684006a90 2024-03-29T11:21:09.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.305+0000 7f169336d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16840cdc40 con 0x7f16840bff20 2024-03-29T11:21:09.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.305+0000 7f169336d640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16840cd620 con 0x7f1684029c50 2024-03-29T11:21:09.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.305+0000 7f169336d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:09.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.305+0000 7f169336d640 10 monclient(hunting): authenticate will time out at 1450.617425s 2024-03-29T11:21:09.288 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.305+0000 7f1690908640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f1684029c50 0x7f16840bf810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:50666/0 (socket says 172.21.15.170:50666) 2024-03-29T11:21:09.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.305+0000 7f1690908640 1 -- 172.21.15.170:0/3274282367 learned_addr learned my addr 172.21.15.170:0/3274282367 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:21:09.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.306+0000 7f1681ffb640 10 client.?.objecter ms_handle_connect 0x7f1684029c50 2024-03-29T11:21:09.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.306+0000 7f1681ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:09.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.307+0000 7f1681ffb640 1 -- 172.21.15.170:0/3274282367 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248439963 0 0) 0x7f16840cd620 con 0x7f1684029c50 2024-03-29T11:21:09.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.307+0000 7f1681ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:09.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.307+0000 7f1681ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:09.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.307+0000 7f1681ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:09.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.307+0000 7f1681ffb640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:21:09.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.307+0000 7f1681ffb640 1 -- 172.21.15.170:0/3274282367 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1670004eb0 con 0x7f1684029c50 2024-03-29T11:21:09.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.308+0000 7f1681ffb640 1 -- 172.21.15.170:0/3274282367 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3749732915 0 0) 0x7f1670004eb0 con 0x7f1684029c50 2024-03-29T11:21:09.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.308+0000 7f1681ffb640 1 -- 172.21.15.170:0/3274282367 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16840cd620 con 0x7f1684029c50 2024-03-29T11:21:09.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.308+0000 7f1681ffb640 1 -- 172.21.15.170:0/3274282367 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f167c004050 con 0x7f1684029c50 2024-03-29T11:21:09.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.308+0000 7f1681ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:09.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.308+0000 7f1681ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:09.289 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.308+0000 7f1681ffb640 10 monclient(hunting): dump: 2024-03-29T11:21:09.290 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:09.290 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:09.290 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:09.290 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:09.290 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:09.290 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:09.290 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:09.290 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:09.290 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:09.290 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:09.290 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.308+0000 7f1681ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:09.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 10 client.?.objecter ms_handle_connect 0x7f1684006a90 2024-03-29T11:21:09.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:09.294 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 10 client.?.objecter ms_handle_connect 0x7f16840bff20 2024-03-29T11:21:09.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:09.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 1 -- 172.21.15.170:0/3274282367 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 881630083 0 0) 0x7f16840cd620 con 0x7f1684029c50 2024-03-29T11:21:09.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 1 -- 172.21.15.170:0/3274282367 >> v1:172.21.15.184:6789/0 conn(0x7f16840bff20 legacy=0x7f16840ca090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:09.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 1 -- 172.21.15.170:0/3274282367 >> v1:172.21.15.170:6789/0 conn(0x7f1684006a90 legacy=0x7f16840ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:09.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:09.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 1 monclient: found mon.c 2024-03-29T11:21:09.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:09.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:09.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 1 -- 172.21.15.170:0/3274282367 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16840cd7b0 con 0x7f1684029c50 2024-03-29T11:21:09.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:09.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:09.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:09.295 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 1 -- 172.21.15.170:0/3274282367 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f167c000ed0 con 0x7f1684029c50 2024-03-29T11:21:09.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:09.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 1 -- 172.21.15.170:0/3274282367 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f167c0012e0 con 0x7f1684029c50 2024-03-29T11:21:09.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:09.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:09.296 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.310+0000 7f1681ffb640 10 monclient: dump: 2024-03-29T11:21:09.296 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:09.296 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:09.296 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:09.296 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:09.296 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:09.296 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:09.296 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:09.296 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:09.296 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:09.296 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:09.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.311+0000 7f169336d640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:21:09.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.311+0000 7f169336d640 10 monclient: _renew_subs 2024-03-29T11:21:09.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.311+0000 7f169336d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:09.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.311+0000 7f169336d640 1 -- 172.21.15.170:0/3274282367 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16840cda00 con 0x7f1684029c50 2024-03-29T11:21:09.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.311+0000 7f169336d640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:09.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.311+0000 7f169336d640 10 monclient: _renew_subs 2024-03-29T11:21:09.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.311+0000 7f169336d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:09.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.311+0000 7f169336d640 1 -- 172.21.15.170:0/3274282367 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16840ca980 con 0x7f1684029c50 2024-03-29T11:21:09.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.311+0000 7f169336d640 1 librados: init done 2024-03-29T11:21:09.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.312+0000 7f16837fe640 4 set_mon_vals no callback set 2024-03-29T11:21:09.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.314+0000 7f169336d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:09.297 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.314+0000 7f169336d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:09.298 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.314+0000 7f169336d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:09.298 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.314+0000 7f169336d640 1 -- 172.21.15.170:0/3274282367 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166c005080 con 0x7f1684029c50 2024-03-29T11:21:09.298 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.314+0000 7f1681ffb640 1 -- 172.21.15.170:0/3274282367 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f167c0014c0 con 0x7f1684029c50 2024-03-29T11:21:09.298 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.315+0000 7f1681ffb640 1 -- 172.21.15.170:0/3274282367 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f167c0849a0 con 0x7f1684029c50 2024-03-29T11:21:09.298 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.315+0000 7f1681ffb640 10 client.4613.objecter ms_dispatch 0x7f168c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:09.298 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.315+0000 7f1681ffb640 3 client.4613.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:09.298 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.315+0000 7f1681ffb640 3 client.4613.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:09.298 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.315+0000 7f1681ffb640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-29T11:21:09.298 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.317+0000 7f1691109640 10 monclient: get_auth_request con 0x7f167006dd60 auth_method 0 2024-03-29T11:21:09.298 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.317+0000 7f1681ffb640 10 client.4613.objecter ms_handle_connect 0x7f167006dd60 2024-03-29T11:21:09.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.323+0000 7f1681ffb640 1 -- 172.21.15.170:0/3274282367 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f167c050280 con 0x7f1684029c50 2024-03-29T11:21:09.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.323+0000 7f1681ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:09.300 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.323+0000 7f1681ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:09.629 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.651+0000 7f169336d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:09.629 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.651+0000 7f169336d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:09.629 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.651+0000 7f169336d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:09.629 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.652+0000 7f169336d640 1 -- 172.21.15.170:0/3274282367 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f166c005330 con 0x7f1684029c50 2024-03-29T11:21:09.630 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.652+0000 7f1681ffb640 1 -- 172.21.15.170:0/3274282367 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f167c0521b0 con 0x7f1684029c50 2024-03-29T11:21:09.630 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.652+0000 7f1681ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:09.630 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.652+0000 7f1681ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:09.630 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:21:09.630 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:09.633 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.656+0000 7f169336d640 10 librados: watch_flush enter 2024-03-29T11:21:09.636 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.659+0000 7f169336d640 10 librados: watch_flush exit 2024-03-29T11:21:09.636 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.659+0000 7f169336d640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-29T11:21:09.636 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.659+0000 7f169336d640 10 client.4613.objecter successfully canceled tick 2024-03-29T11:21:09.636 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.659+0000 7f169336d640 1 -- 172.21.15.170:0/3274282367 >> v1:172.21.15.170:6816/3298994444 conn(0x7f167006dd60 legacy=0x7f1670070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:09.636 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.659+0000 7f169336d640 10 monclient: shutdown 2024-03-29T11:21:09.636 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.659+0000 7f169336d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:09.636 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.659+0000 7f169336d640 1 -- 172.21.15.170:0/3274282367 >> v1:172.21.15.170:6790/0 conn(0x7f1684029c50 legacy=0x7f16840bf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:09.637 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.659+0000 7f169336d640 1 -- 172.21.15.170:0/3274282367 shutdown_connections 2024-03-29T11:21:09.637 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.659+0000 7f169336d640 1 -- 172.21.15.170:0/3274282367 >> 172.21.15.170:0/3274282367 conn(0x7f1684005910 msgr2=0x7f16840c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:09.637 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.660+0000 7f169336d640 1 -- 172.21.15.170:0/3274282367 shutdown_connections 2024-03-29T11:21:09.637 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.660+0000 7f169336d640 1 -- 172.21.15.170:0/3274282367 wait complete. 2024-03-29T11:21:09.637 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:09.660+0000 7f169336d640 1 librados: shutdown 2024-03-29T11:21:09.659 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:16.467 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:16.620 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.644+0000 7f1b1a645640 10 monclient: get_monmap_and_config 2024-03-29T11:21:16.620 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.644+0000 7f1b1a645640 10 monclient: build_initial_monmap 2024-03-29T11:21:16.620 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.644+0000 7f1b1a645640 10 monclient: monmap: 2024-03-29T11:21:16.620 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:21:16.620 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:16.620 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:21:16.644686+0000 2024-03-29T11:21:16.620 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:21:16.644686+0000 2024-03-29T11:21:16.620 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:16.620 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:16.621 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:21:16.621 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:21:16.621 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:21:16.621 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:16.621 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.645+0000 7f1b1a645640 1 Processor -- start 2024-03-29T11:21:16.624 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.648+0000 7f1b1a645640 1 -- start start 2024-03-29T11:21:16.624 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.648+0000 7f1b1a645640 10 monclient: init 2024-03-29T11:21:16.625 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.649+0000 7f1b1a645640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:16.625 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.649+0000 7f1b1a645640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:16.625 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.649+0000 7f1b1a645640 10 monclient(hunting): picked mon.noname-b con 0x7f1b1406bef0 addr v1:172.21.15.170:6790/0 2024-03-29T11:21:16.625 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.649+0000 7f1b1a645640 10 monclient(hunting): picked mon.noname-c con 0x7f1b14071ef0 addr v1:172.21.15.184:6789/0 2024-03-29T11:21:16.625 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.649+0000 7f1b1a645640 10 monclient(hunting): picked mon.noname-a con 0x7f1b140afd40 addr v1:172.21.15.170:6789/0 2024-03-29T11:21:16.625 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.649+0000 7f1b1a645640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1406ceb0 con 0x7f1b140afd40 2024-03-29T11:21:16.625 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.649+0000 7f1b1a645640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1406dbb0 con 0x7f1b14071ef0 2024-03-29T11:21:16.625 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.649+0000 7f1b1a645640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1406d340 con 0x7f1b1406bef0 2024-03-29T11:21:16.626 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.649+0000 7f1b1a645640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:16.626 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.649+0000 7f1b1a645640 10 monclient(hunting): authenticate will time out at 1457.960613s 2024-03-29T11:21:16.628 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.652+0000 7f1b13fff640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f1b1406bef0 0x7f1b140717e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:46409/0 (socket says 172.21.15.170:46409) 2024-03-29T11:21:16.628 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.653+0000 7f1b13fff640 1 -- 172.21.15.170:0/1942703946 learned_addr learned my addr 172.21.15.170:0/1942703946 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:21:16.631 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.654+0000 7f1b12ffd640 1 -- 172.21.15.170:0/1942703946 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26311818 0 0) 0x7f1b1406dbb0 con 0x7f1b14071ef0 2024-03-29T11:21:16.631 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.654+0000 7f1b12ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:16.631 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.654+0000 7f1b12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:16.631 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.654+0000 7f1b12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:16.631 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.654+0000 7f1b12ffd640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:21:16.631 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.654+0000 7f1b12ffd640 1 -- 172.21.15.170:0/1942703946 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b04003520 con 0x7f1b14071ef0 2024-03-29T11:21:16.631 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.654+0000 7f1b12ffd640 1 -- 172.21.15.170:0/1942703946 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3812787468 0 0) 0x7f1b04003520 con 0x7f1b14071ef0 2024-03-29T11:21:16.631 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.654+0000 7f1b12ffd640 1 -- 172.21.15.170:0/1942703946 >> v1:172.21.15.170:6790/0 conn(0x7f1b1406bef0 legacy=0x7f1b140717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:16.631 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.654+0000 7f1b12ffd640 1 -- 172.21.15.170:0/1942703946 >> v1:172.21.15.170:6789/0 conn(0x7f1b140afd40 legacy=0x7f1b14151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:16.631 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.655+0000 7f1b12ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:16.631 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.655+0000 7f1b12ffd640 1 monclient: found mon.noname-c 2024-03-29T11:21:16.631 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.655+0000 7f1b12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:16.633 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.655+0000 7f1b12ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:21:16.633 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.655+0000 7f1b12ffd640 1 -- 172.21.15.170:0/1942703946 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b1406fca0 con 0x7f1b14071ef0 2024-03-29T11:21:16.633 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.655+0000 7f1b12ffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:16.633 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.655+0000 7f1b12ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:16.633 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.655+0000 7f1b12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:16.633 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.655+0000 7f1b12ffd640 1 -- 172.21.15.170:0/1942703946 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f1b08001690 con 0x7f1b14071ef0 2024-03-29T11:21:16.633 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.655+0000 7f1b12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:16.633 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.655+0000 7f1b12ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:16.633 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.655+0000 7f1b12ffd640 10 monclient: dump: 2024-03-29T11:21:16.634 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:16.634 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:16.634 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:16.634 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:16.634 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:16.634 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:16.634 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:16.634 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:16.634 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:16.634 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:16.634 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.655+0000 7f1b1a645640 5 monclient: authenticate success, global_id 4531 2024-03-29T11:21:16.634 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.655+0000 7f1b1a645640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:16.634 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.656+0000 7f1b12ffd640 1 -- 172.21.15.170:0/1942703946 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b08001860 con 0x7f1b14071ef0 2024-03-29T11:21:16.634 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.656+0000 7f1b12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:16.635 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.656+0000 7f1b1a645640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:16.635 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.656+0000 7f1b1a645640 4 set_mon_vals no callback set 2024-03-29T11:21:16.635 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.657+0000 7f1b1a645640 10 monclient: shutdown 2024-03-29T11:21:16.635 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.657+0000 7f1b1a645640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:16.635 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.657+0000 7f1b1a645640 1 -- 172.21.15.170:0/1942703946 >> v1:172.21.15.184:6789/0 conn(0x7f1b14071ef0 legacy=0x7f1b14146390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:16.635 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.657+0000 7f1b1a645640 1 -- 172.21.15.170:0/1942703946 shutdown_connections 2024-03-29T11:21:16.635 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.657+0000 7f1b1a645640 1 -- 172.21.15.170:0/1942703946 >> 172.21.15.170:0/1942703946 conn(0x7f1b140671f0 msgr2=0x7f1b140b59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:16.636 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.658+0000 7f1b1a645640 1 -- 172.21.15.170:0/1942703946 shutdown_connections 2024-03-29T11:21:16.636 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.658+0000 7f1b1a645640 1 -- 172.21.15.170:0/1942703946 wait complete. 2024-03-29T11:21:16.636 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.659+0000 7f1b1a645640 10 monclient: build_initial_monmap 2024-03-29T11:21:16.636 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.659+0000 7f1b1a645640 10 monclient: monmap: 2024-03-29T11:21:16.637 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:21:16.637 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:16.637 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:21:16.637 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:21:16.637 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:16.637 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:16.637 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:21:16.637 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:21:16.637 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:21:16.637 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:16.637 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.660+0000 7f1b1a645640 1 librados: starting msgr at 2024-03-29T11:21:16.637 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.660+0000 7f1b1a645640 1 librados: starting objecter 2024-03-29T11:21:16.637 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.660+0000 7f1b1a645640 1 Processor -- start 2024-03-29T11:21:16.637 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.660+0000 7f1b1a645640 1 -- start start 2024-03-29T11:21:16.637 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.661+0000 7f1b1a645640 1 librados: setting wanted keys 2024-03-29T11:21:16.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.661+0000 7f1b1a645640 1 librados: calling monclient init 2024-03-29T11:21:16.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.661+0000 7f1b1a645640 10 monclient: init 2024-03-29T11:21:16.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.661+0000 7f1b1a645640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:16.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.661+0000 7f1b1a645640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:16.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.662+0000 7f1b1a645640 10 monclient(hunting): picked mon.noname-b con 0x7f1b1406bef0 addr v1:172.21.15.184:6789/0 2024-03-29T11:21:16.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.662+0000 7f1b1a645640 10 monclient(hunting): picked mon.noname-a con 0x7f1b14071ef0 addr v1:172.21.15.170:6789/0 2024-03-29T11:21:16.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.662+0000 7f1b1a645640 10 monclient(hunting): picked mon.noname-c con 0x7f1b140afd40 addr v1:172.21.15.170:6790/0 2024-03-29T11:21:16.639 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.662+0000 7f1b1a645640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b141257b0 con 0x7f1b14071ef0 2024-03-29T11:21:16.640 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.662+0000 7f1b1a645640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b140aca30 con 0x7f1b1406bef0 2024-03-29T11:21:16.640 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.663+0000 7f1b1a645640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b140aea40 con 0x7f1b140afd40 2024-03-29T11:21:16.640 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.663+0000 7f1b1a645640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:16.640 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.663+0000 7f1b1a645640 10 monclient(hunting): authenticate will time out at 1457.974219s 2024-03-29T11:21:16.642 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.666+0000 7f1b18be2640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f1b140afd40 0x7f1b141ee440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:46489/0 (socket says 172.21.15.170:46489) 2024-03-29T11:21:16.642 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.666+0000 7f1b18be2640 1 -- 172.21.15.170:0/2346056315 learned_addr learned my addr 172.21.15.170:0/2346056315 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:21:16.647 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.669+0000 7f1b10ff9640 10 client.?.objecter ms_handle_connect 0x7f1b140afd40 2024-03-29T11:21:16.647 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.669+0000 7f1b10ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:16.647 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.669+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40313826 0 0) 0x7f1b140aea40 con 0x7f1b140afd40 2024-03-29T11:21:16.647 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.669+0000 7f1b10ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:16.647 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.669+0000 7f1b10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:16.647 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.669+0000 7f1b10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:16.647 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.669+0000 7f1b10ff9640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:21:16.647 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.669+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af8004eb0 con 0x7f1b140afd40 2024-03-29T11:21:16.647 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.669+0000 7f1b10ff9640 10 client.?.objecter ms_handle_connect 0x7f1b1406bef0 2024-03-29T11:21:16.647 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.669+0000 7f1b10ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:16.648 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.669+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798355598 0 0) 0x7f1b140aca30 con 0x7f1b1406bef0 2024-03-29T11:21:16.648 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.670+0000 7f1b10ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:16.648 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.670+0000 7f1b10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:16.648 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.670+0000 7f1b10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:16.648 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.670+0000 7f1b10ff9640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:21:16.648 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.670+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b140aea40 con 0x7f1b1406bef0 2024-03-29T11:21:16.648 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.670+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2209025236 0 0) 0x7f1af8004eb0 con 0x7f1b140afd40 2024-03-29T11:21:16.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.670+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1af8005490 con 0x7f1b140afd40 2024-03-29T11:21:16.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.670+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f1b0c001cc0 con 0x7f1b140afd40 2024-03-29T11:21:16.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.670+0000 7f1b10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:16.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.670+0000 7f1b10ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:16.649 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.670+0000 7f1b10ff9640 10 monclient(hunting): dump: 2024-03-29T11:21:16.649 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:16.649 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:16.649 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:16.649 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:16.649 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:16.649 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:16.649 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:16.649 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:16.650 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:16.650 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:16.650 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.670+0000 7f1b10ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:16.650 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.671+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1963870802 0 0) 0x7f1b140aea40 con 0x7f1b1406bef0 2024-03-29T11:21:16.650 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.671+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1af8005b60 con 0x7f1b1406bef0 2024-03-29T11:21:16.650 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.671+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f1b000014d0 con 0x7f1b1406bef0 2024-03-29T11:21:16.650 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.671+0000 7f1b10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:16.650 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.671+0000 7f1b10ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:16.650 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.671+0000 7f1b10ff9640 10 monclient(hunting): dump: 2024-03-29T11:21:16.650 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:16.650 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:16.650 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:16.650 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:16.651 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:16.651 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:16.651 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:16.651 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:16.651 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:16.651 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:16.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.671+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3189142464 0 0) 0x7f1af8005b60 con 0x7f1b1406bef0 2024-03-29T11:21:16.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.671+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 >> v1:172.21.15.170:6790/0 conn(0x7f1b140afd40 legacy=0x7f1b141ee440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:16.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.671+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 >> v1:172.21.15.170:6789/0 conn(0x7f1b14071ef0 legacy=0x7f1b140ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:16.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.671+0000 7f1b10ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:21:16.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.671+0000 7f1b10ff9640 1 monclient: found mon.b 2024-03-29T11:21:16.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.671+0000 7f1b10ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:16.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.671+0000 7f1b10ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:21:16.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.671+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b140aed20 con 0x7f1b1406bef0 2024-03-29T11:21:16.651 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.671+0000 7f1b10ff9640 10 monclient: _finish_auth 0 2024-03-29T11:21:16.652 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.671+0000 7f1b10ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:21:16.652 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.671+0000 7f1b10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:16.652 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.672+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b00001a00 con 0x7f1b1406bef0 2024-03-29T11:21:16.652 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.672+0000 7f1b10ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:16.652 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.672+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f1b000014d0 con 0x7f1b1406bef0 2024-03-29T11:21:16.652 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.672+0000 7f1b10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:16.652 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.672+0000 7f1b10ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:16.652 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.672+0000 7f1b10ff9640 10 monclient: dump: 2024-03-29T11:21:16.652 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:16.652 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:16.652 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:16.652 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:16.652 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:16.652 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:16.652 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:16.653 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:16.653 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:16.653 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:16.653 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.672+0000 7f1b127fc640 4 set_mon_vals no callback set 2024-03-29T11:21:16.654 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.673+0000 7f1b1a645640 5 monclient: authenticate success, global_id 4534 2024-03-29T11:21:16.654 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.673+0000 7f1b1a645640 10 monclient: _renew_subs 2024-03-29T11:21:16.654 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.673+0000 7f1b1a645640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:21:16.654 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.674+0000 7f1b1a645640 1 -- 172.21.15.170:0/2346056315 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b140adc40 con 0x7f1b1406bef0 2024-03-29T11:21:16.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.674+0000 7f1b1a645640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:16.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.674+0000 7f1b1a645640 10 monclient: _renew_subs 2024-03-29T11:21:16.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.674+0000 7f1b1a645640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:21:16.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.674+0000 7f1b1a645640 1 -- 172.21.15.170:0/2346056315 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b140af1b0 con 0x7f1b1406bef0 2024-03-29T11:21:16.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.674+0000 7f1b1a645640 1 librados: init done 2024-03-29T11:21:16.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.676+0000 7f1b1a645640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:16.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.676+0000 7f1b1a645640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:16.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.676+0000 7f1b1a645640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:21:16.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.676+0000 7f1b1a645640 1 -- 172.21.15.170:0/2346056315 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b140ac890 con 0x7f1b1406bef0 2024-03-29T11:21:16.655 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.678+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f1b00001bf0 con 0x7f1b1406bef0 2024-03-29T11:21:16.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.679+0000 7f1b137fe640 10 monclient: get_auth_request con 0x7f1af806e5f0 auth_method 0 2024-03-29T11:21:16.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.679+0000 7f1b10ff9640 10 client.4534.objecter ms_handle_connect 0x7f1af806e5f0 2024-03-29T11:21:16.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.679+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f1b000543e0 con 0x7f1b1406bef0 2024-03-29T11:21:16.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.679+0000 7f1b10ff9640 10 client.4534.objecter ms_dispatch 0x7f1b14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:16.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.679+0000 7f1b10ff9640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:16.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.679+0000 7f1b10ff9640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:16.656 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.679+0000 7f1b10ff9640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-29T11:21:16.665 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.688+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b00050680 con 0x7f1b1406bef0 2024-03-29T11:21:16.665 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.688+0000 7f1b10ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:16.665 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:16.688+0000 7f1b10ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:16.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.016+0000 7f1b1a645640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:16.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.016+0000 7f1b1a645640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:16.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.016+0000 7f1b1a645640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:21:16.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.016+0000 7f1b1a645640 1 -- 172.21.15.170:0/2346056315 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b140ac890 con 0x7f1b1406bef0 2024-03-29T11:21:16.995 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:21:16.995 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:16.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.018+0000 7f1b10ff9640 1 -- 172.21.15.170:0/2346056315 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f1b00050680 con 0x7f1b1406bef0 2024-03-29T11:21:16.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.018+0000 7f1b10ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:16.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.018+0000 7f1b10ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:16.997 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.021+0000 7f1b1a645640 10 librados: watch_flush enter 2024-03-29T11:21:17.000 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.024+0000 7f1b1a645640 10 librados: watch_flush exit 2024-03-29T11:21:17.000 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.024+0000 7f1b1a645640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-29T11:21:17.000 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.024+0000 7f1b1a645640 10 client.4534.objecter successfully canceled tick 2024-03-29T11:21:17.003 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.027+0000 7f1b1a645640 1 -- 172.21.15.170:0/2346056315 >> v1:172.21.15.170:6816/3298994444 conn(0x7f1af806e5f0 legacy=0x7f1af8070ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:17.003 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.027+0000 7f1b1a645640 10 monclient: shutdown 2024-03-29T11:21:17.003 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.027+0000 7f1b1a645640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:17.003 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.027+0000 7f1b1a645640 1 -- 172.21.15.170:0/2346056315 >> v1:172.21.15.184:6789/0 conn(0x7f1b1406bef0 legacy=0x7f1b141250a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:17.004 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.027+0000 7f1b1a645640 1 -- 172.21.15.170:0/2346056315 shutdown_connections 2024-03-29T11:21:17.004 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.027+0000 7f1b1a645640 1 -- 172.21.15.170:0/2346056315 >> 172.21.15.170:0/2346056315 conn(0x7f1b140671f0 msgr2=0x7f1b140b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:17.004 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.028+0000 7f1b1a645640 1 -- 172.21.15.170:0/2346056315 shutdown_connections 2024-03-29T11:21:17.004 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.028+0000 7f1b1a645640 1 -- 172.21.15.170:0/2346056315 wait complete. 2024-03-29T11:21:17.004 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:17.028+0000 7f1b1a645640 1 librados: shutdown 2024-03-29T11:21:17.022 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:23.829 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:23.972 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.996+0000 7fc135f32640 10 monclient: get_monmap_and_config 2024-03-29T11:21:23.972 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.996+0000 7fc135f32640 10 monclient: build_initial_monmap 2024-03-29T11:21:23.972 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.996+0000 7fc135f32640 10 monclient: monmap: 2024-03-29T11:21:23.972 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:21:23.972 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:23.972 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:21:23.996820+0000 2024-03-29T11:21:23.972 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:21:23.996820+0000 2024-03-29T11:21:23.972 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:23.973 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:23.973 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:21:23.973 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:21:23.973 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:21:23.973 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:23.973 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.997+0000 7fc135f32640 1 Processor -- start 2024-03-29T11:21:23.973 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.997+0000 7fc135f32640 1 -- start start 2024-03-29T11:21:23.973 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.997+0000 7fc135f32640 10 monclient: init 2024-03-29T11:21:23.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.998+0000 7fc135f32640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:23.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.998+0000 7fc135f32640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:23.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.998+0000 7fc135f32640 10 monclient(hunting): picked mon.noname-b con 0x7fc1301457f0 addr v1:172.21.15.170:6790/0 2024-03-29T11:21:23.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.998+0000 7fc135f32640 10 monclient(hunting): picked mon.noname-a con 0x7fc13013f650 addr v1:172.21.15.170:6789/0 2024-03-29T11:21:23.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.998+0000 7fc135f32640 10 monclient(hunting): picked mon.noname-c con 0x7fc1300a9d70 addr v1:172.21.15.184:6789/0 2024-03-29T11:21:23.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.998+0000 7fc135f32640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1300aaef0 con 0x7fc13013f650 2024-03-29T11:21:23.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.998+0000 7fc135f32640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc130145bd0 con 0x7fc1300a9d70 2024-03-29T11:21:23.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.998+0000 7fc135f32640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc130145eb0 con 0x7fc1301457f0 2024-03-29T11:21:23.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.998+0000 7fc135f32640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:23.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.998+0000 7fc135f32640 10 monclient(hunting): authenticate will time out at 1465.309647s 2024-03-29T11:21:23.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.998+0000 7fc12f7fe640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fc1301457f0 0x7fc13013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:39338/0 (socket says 172.21.15.170:39338) 2024-03-29T11:21:23.975 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.998+0000 7fc12f7fe640 1 -- 172.21.15.170:0/555772332 learned_addr learned my addr 172.21.15.170:0/555772332 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:21:23.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.999+0000 7fc12e7fc640 1 -- 172.21.15.170:0/555772332 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622129677 0 0) 0x7fc130145eb0 con 0x7fc1301457f0 2024-03-29T11:21:23.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.999+0000 7fc12e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:23.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.999+0000 7fc12e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:23.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.999+0000 7fc12e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:23.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.999+0000 7fc12e7fc640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:21:23.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.999+0000 7fc12e7fc640 1 -- 172.21.15.170:0/555772332 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc11c003520 con 0x7fc1301457f0 2024-03-29T11:21:23.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.999+0000 7fc12e7fc640 1 -- 172.21.15.170:0/555772332 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3690182988 0 0) 0x7fc11c003520 con 0x7fc1301457f0 2024-03-29T11:21:23.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.999+0000 7fc12e7fc640 1 -- 172.21.15.170:0/555772332 >> v1:172.21.15.184:6789/0 conn(0x7fc1300a9d70 legacy=0x7fc13014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:23.976 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:23.999+0000 7fc12e7fc640 1 -- 172.21.15.170:0/555772332 >> v1:172.21.15.170:6789/0 conn(0x7fc13013f650 legacy=0x7fc1300b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:23.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.001+0000 7fc12e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:23.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.001+0000 7fc12e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:21:23.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.001+0000 7fc12e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:23.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.001+0000 7fc12e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:21:23.977 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.001+0000 7fc12e7fc640 1 -- 172.21.15.170:0/555772332 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc130146340 con 0x7fc1301457f0 2024-03-29T11:21:23.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.001+0000 7fc12e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:23.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.001+0000 7fc12e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:23.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.001+0000 7fc12e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:23.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.001+0000 7fc12e7fc640 1 -- 172.21.15.170:0/555772332 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fc128001690 con 0x7fc1301457f0 2024-03-29T11:21:23.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.001+0000 7fc12e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:23.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.001+0000 7fc12e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:23.978 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.001+0000 7fc12e7fc640 10 monclient: dump: 2024-03-29T11:21:23.978 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:23.978 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:23.978 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:23.979 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:23.979 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:23.979 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:23.979 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:23.979 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:23.979 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:23.979 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:23.979 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.001+0000 7fc135f32640 5 monclient: authenticate success, global_id 4619 2024-03-29T11:21:23.979 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.001+0000 7fc135f32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:23.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.002+0000 7fc12e7fc640 1 -- 172.21.15.170:0/555772332 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc128001920 con 0x7fc1301457f0 2024-03-29T11:21:23.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.002+0000 7fc12e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:23.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.002+0000 7fc135f32640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:23.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.002+0000 7fc135f32640 4 set_mon_vals no callback set 2024-03-29T11:21:23.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.003+0000 7fc135f32640 10 monclient: shutdown 2024-03-29T11:21:23.981 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.003+0000 7fc135f32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:23.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.003+0000 7fc135f32640 1 -- 172.21.15.170:0/555772332 >> v1:172.21.15.170:6790/0 conn(0x7fc1301457f0 legacy=0x7fc13013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:23.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.003+0000 7fc135f32640 1 -- 172.21.15.170:0/555772332 shutdown_connections 2024-03-29T11:21:23.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.003+0000 7fc135f32640 1 -- 172.21.15.170:0/555772332 >> 172.21.15.170:0/555772332 conn(0x7fc130066a00 msgr2=0x7fc130066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:23.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.003+0000 7fc135f32640 1 -- 172.21.15.170:0/555772332 shutdown_connections 2024-03-29T11:21:23.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.004+0000 7fc135f32640 1 -- 172.21.15.170:0/555772332 wait complete. 2024-03-29T11:21:23.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.004+0000 7fc135f32640 10 monclient: build_initial_monmap 2024-03-29T11:21:23.982 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.004+0000 7fc135f32640 10 monclient: monmap: 2024-03-29T11:21:23.983 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:21:23.983 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:23.983 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:21:23.983 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:21:23.983 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:23.983 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:23.983 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:21:23.983 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:21:23.983 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:21:23.984 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:23.984 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.005+0000 7fc135f32640 1 librados: starting msgr at 2024-03-29T11:21:23.984 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.005+0000 7fc135f32640 1 librados: starting objecter 2024-03-29T11:21:23.984 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.005+0000 7fc135f32640 1 Processor -- start 2024-03-29T11:21:23.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.006+0000 7fc135f32640 1 -- start start 2024-03-29T11:21:23.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.006+0000 7fc135f32640 1 librados: setting wanted keys 2024-03-29T11:21:23.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.006+0000 7fc135f32640 1 librados: calling monclient init 2024-03-29T11:21:23.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.006+0000 7fc135f32640 10 monclient: init 2024-03-29T11:21:23.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc135f32640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:23.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc135f32640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:23.988 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc135f32640 10 monclient(hunting): picked mon.noname-c con 0x7fc1300a9d70 addr v1:172.21.15.170:6790/0 2024-03-29T11:21:23.989 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc135f32640 10 monclient(hunting): picked mon.noname-a con 0x7fc13013f650 addr v1:172.21.15.170:6789/0 2024-03-29T11:21:23.989 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc135f32640 10 monclient(hunting): picked mon.noname-b con 0x7fc1301457f0 addr v1:172.21.15.184:6789/0 2024-03-29T11:21:23.989 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc135f32640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1301dc160 con 0x7fc13013f650 2024-03-29T11:21:23.989 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc135f32640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1301dc5f0 con 0x7fc1301457f0 2024-03-29T11:21:23.989 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc135f32640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc13006ecd0 con 0x7fc1300a9d70 2024-03-29T11:21:23.989 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc135f32640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:23.989 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc135f32640 10 monclient(hunting): authenticate will time out at 1465.318225s 2024-03-29T11:21:23.989 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc12ffff640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fc1301457f0 0x7fc1301e6100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:50482/0 (socket says 172.21.15.170:50482) 2024-03-29T11:21:23.989 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc12ffff640 1 -- 172.21.15.170:0/1673538894 learned_addr learned my addr 172.21.15.170:0/1673538894 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:21:23.989 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc127fff640 10 client.?.objecter ms_handle_connect 0x7fc1300a9d70 2024-03-29T11:21:23.989 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc127fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:23.989 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc127fff640 1 -- 172.21.15.170:0/1673538894 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106879501 0 0) 0x7fc13006ecd0 con 0x7fc1300a9d70 2024-03-29T11:21:23.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc127fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:23.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc127fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:23.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc127fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:23.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc127fff640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:21:23.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.007+0000 7fc127fff640 1 -- 172.21.15.170:0/1673538894 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc110004eb0 con 0x7fc1300a9d70 2024-03-29T11:21:23.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.008+0000 7fc127fff640 1 -- 172.21.15.170:0/1673538894 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3717874240 0 0) 0x7fc110004eb0 con 0x7fc1300a9d70 2024-03-29T11:21:23.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.008+0000 7fc127fff640 1 -- 172.21.15.170:0/1673538894 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc13006ecd0 con 0x7fc1300a9d70 2024-03-29T11:21:23.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.008+0000 7fc127fff640 10 client.?.objecter ms_handle_connect 0x7fc1301457f0 2024-03-29T11:21:23.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.008+0000 7fc127fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:23.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.008+0000 7fc127fff640 1 -- 172.21.15.170:0/1673538894 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fc128004920 con 0x7fc1300a9d70 2024-03-29T11:21:23.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.008+0000 7fc127fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:23.990 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.008+0000 7fc127fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:23.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.008+0000 7fc127fff640 10 monclient(hunting): dump: 2024-03-29T11:21:23.991 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:23.991 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:23.991 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:23.991 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:23.991 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:23.991 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:23.991 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:23.991 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:23.991 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:23.991 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:23.991 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.008+0000 7fc127fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:23.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.008+0000 7fc127fff640 1 -- 172.21.15.170:0/1673538894 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772729158 0 0) 0x7fc1301dc5f0 con 0x7fc1301457f0 2024-03-29T11:21:23.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.008+0000 7fc127fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:23.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.008+0000 7fc127fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:23.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.008+0000 7fc127fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:23.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.008+0000 7fc127fff640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:21:23.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.008+0000 7fc127fff640 1 -- 172.21.15.170:0/1673538894 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc110004eb0 con 0x7fc1301457f0 2024-03-29T11:21:23.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.008+0000 7fc127fff640 1 -- 172.21.15.170:0/1673538894 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4012032164 0 0) 0x7fc13006ecd0 con 0x7fc1300a9d70 2024-03-29T11:21:23.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.008+0000 7fc127fff640 1 -- 172.21.15.170:0/1673538894 >> v1:172.21.15.184:6789/0 conn(0x7fc1301457f0 legacy=0x7fc1301e6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:23.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.008+0000 7fc127fff640 1 -- 172.21.15.170:0/1673538894 >> v1:172.21.15.170:6789/0 conn(0x7fc13013f650 legacy=0x7fc13013fda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:23.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.009+0000 7fc127fff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:23.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.009+0000 7fc127fff640 1 monclient: found mon.c 2024-03-29T11:21:23.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.009+0000 7fc127fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:23.992 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.009+0000 7fc127fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:23.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.009+0000 7fc127fff640 1 -- 172.21.15.170:0/1673538894 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1301dbdd0 con 0x7fc1300a9d70 2024-03-29T11:21:23.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.009+0000 7fc127fff640 10 monclient: _finish_auth 0 2024-03-29T11:21:23.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.009+0000 7fc127fff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:23.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.009+0000 7fc127fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:23.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.009+0000 7fc135f32640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:21:23.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.009+0000 7fc135f32640 10 monclient: _renew_subs 2024-03-29T11:21:23.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.009+0000 7fc135f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:23.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.009+0000 7fc135f32640 1 -- 172.21.15.170:0/1673538894 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13013b030 con 0x7fc1300a9d70 2024-03-29T11:21:23.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.010+0000 7fc135f32640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:23.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.010+0000 7fc135f32640 10 monclient: _renew_subs 2024-03-29T11:21:23.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.010+0000 7fc135f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:23.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.010+0000 7fc135f32640 1 -- 172.21.15.170:0/1673538894 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1301404b0 con 0x7fc1300a9d70 2024-03-29T11:21:23.993 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.010+0000 7fc135f32640 1 librados: init done 2024-03-29T11:21:23.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.011+0000 7fc135f32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:23.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.011+0000 7fc135f32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:23.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.011+0000 7fc135f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:23.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.011+0000 7fc135f32640 1 -- 172.21.15.170:0/1673538894 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc130075f10 con 0x7fc1300a9d70 2024-03-29T11:21:23.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.012+0000 7fc127fff640 1 -- 172.21.15.170:0/1673538894 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc128001b90 con 0x7fc1300a9d70 2024-03-29T11:21:23.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.012+0000 7fc127fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:23.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.012+0000 7fc127fff640 1 -- 172.21.15.170:0/1673538894 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fc128004920 con 0x7fc1300a9d70 2024-03-29T11:21:23.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.012+0000 7fc127fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:23.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.012+0000 7fc127fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:23.994 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.012+0000 7fc127fff640 10 monclient: dump: 2024-03-29T11:21:23.994 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:23.994 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:23.995 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:23.995 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:23.995 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:23.995 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:23.995 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:23.995 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:23.995 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:23.995 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:23.995 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.012+0000 7fc127fff640 1 -- 172.21.15.170:0/1673538894 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fc128004db0 con 0x7fc1300a9d70 2024-03-29T11:21:23.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.012+0000 7fc12dffb640 4 set_mon_vals no callback set 2024-03-29T11:21:23.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.013+0000 7fc127fff640 1 -- 172.21.15.170:0/1673538894 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fc128084a90 con 0x7fc1300a9d70 2024-03-29T11:21:23.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.013+0000 7fc127fff640 10 client.4622.objecter ms_dispatch 0x7fc130000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:23.996 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.013+0000 7fc127fff640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:23.997 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.013+0000 7fc127fff640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:23.997 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.013+0000 7fc127fff640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-29T11:21:23.997 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.013+0000 7fc12effd640 10 monclient: get_auth_request con 0x7fc11006dd90 auth_method 0 2024-03-29T11:21:23.997 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.013+0000 7fc127fff640 10 client.4622.objecter ms_handle_connect 0x7fc11006dd90 2024-03-29T11:21:23.997 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.019+0000 7fc127fff640 1 -- 172.21.15.170:0/1673538894 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc1280504b0 con 0x7fc1300a9d70 2024-03-29T11:21:23.997 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.019+0000 7fc127fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:23.997 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.019+0000 7fc127fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:24.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.328+0000 7fc135f32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:24.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.328+0000 7fc135f32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:24.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.328+0000 7fc135f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:24.305 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.328+0000 7fc135f32640 1 -- 172.21.15.170:0/1673538894 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1300b3680 con 0x7fc1300a9d70 2024-03-29T11:21:24.306 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.330+0000 7fc127fff640 1 -- 172.21.15.170:0/1673538894 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fc1280523e0 con 0x7fc1300a9d70 2024-03-29T11:21:24.306 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.330+0000 7fc127fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:24.306 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.330+0000 7fc127fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:24.306 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:21:24.307 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:24.310 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.333+0000 7fc135f32640 10 librados: watch_flush enter 2024-03-29T11:21:24.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.334+0000 7fc135f32640 10 librados: watch_flush exit 2024-03-29T11:21:24.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.334+0000 7fc135f32640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-29T11:21:24.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.334+0000 7fc135f32640 10 client.4622.objecter successfully canceled tick 2024-03-29T11:21:24.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.334+0000 7fc135f32640 1 -- 172.21.15.170:0/1673538894 >> v1:172.21.15.170:6816/3298994444 conn(0x7fc11006dd90 legacy=0x7fc110070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:24.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.334+0000 7fc135f32640 10 monclient: shutdown 2024-03-29T11:21:24.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.334+0000 7fc135f32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:24.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.334+0000 7fc135f32640 1 -- 172.21.15.170:0/1673538894 >> v1:172.21.15.170:6790/0 conn(0x7fc1300a9d70 legacy=0x7fc1301db6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:24.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.334+0000 7fc135f32640 1 -- 172.21.15.170:0/1673538894 shutdown_connections 2024-03-29T11:21:24.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.334+0000 7fc135f32640 1 -- 172.21.15.170:0/1673538894 >> 172.21.15.170:0/1673538894 conn(0x7fc130066a00 msgr2=0x7fc1300b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:24.311 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.334+0000 7fc135f32640 1 -- 172.21.15.170:0/1673538894 shutdown_connections 2024-03-29T11:21:24.312 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.335+0000 7fc135f32640 1 -- 172.21.15.170:0/1673538894 wait complete. 2024-03-29T11:21:24.312 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:24.336+0000 7fc135f32640 1 librados: shutdown 2024-03-29T11:21:24.330 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:31.136 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:31.319 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.343+0000 7f562f3fb640 10 monclient: get_monmap_and_config 2024-03-29T11:21:31.319 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.343+0000 7f562f3fb640 10 monclient: build_initial_monmap 2024-03-29T11:21:31.319 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.343+0000 7f562f3fb640 10 monclient: monmap: 2024-03-29T11:21:31.319 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:21:31.320 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:31.320 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:21:31.343683+0000 2024-03-29T11:21:31.320 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:21:31.343683+0000 2024-03-29T11:21:31.320 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:31.320 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:31.320 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:21:31.320 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:21:31.320 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:21:31.320 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:31.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.344+0000 7f562f3fb640 1 Processor -- start 2024-03-29T11:21:31.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.344+0000 7f562f3fb640 1 -- start start 2024-03-29T11:21:31.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.344+0000 7f562f3fb640 10 monclient: init 2024-03-29T11:21:31.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.345+0000 7f562f3fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:31.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.345+0000 7f562f3fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:31.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.345+0000 7f562f3fb640 10 monclient(hunting): picked mon.noname-b con 0x7f56280b0500 addr v1:172.21.15.170:6790/0 2024-03-29T11:21:31.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.346+0000 7f562f3fb640 10 monclient(hunting): picked mon.noname-a con 0x7f56280b08e0 addr v1:172.21.15.170:6789/0 2024-03-29T11:21:31.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.346+0000 7f562f3fb640 10 monclient(hunting): picked mon.noname-c con 0x7f56280aeca0 addr v1:172.21.15.184:6789/0 2024-03-29T11:21:31.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.346+0000 7f562f3fb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5628140620 con 0x7f56280b08e0 2024-03-29T11:21:31.322 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.346+0000 7f562f3fb640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562813daf0 con 0x7f56280aeca0 2024-03-29T11:21:31.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.346+0000 7f562f3fb640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56280ae180 con 0x7f56280b0500 2024-03-29T11:21:31.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.346+0000 7f562f3fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:31.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.346+0000 7f562f3fb640 10 monclient(hunting): authenticate will time out at 1472.657377s 2024-03-29T11:21:31.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.346+0000 7f562d197640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f56280b0500 0x7f562813ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:39822/0 (socket says 172.21.15.170:39822) 2024-03-29T11:21:31.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.346+0000 7f562d197640 1 -- 172.21.15.170:0/4262698296 learned_addr learned my addr 172.21.15.170:0/4262698296 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:21:31.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 1 -- 172.21.15.170:0/4262698296 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532460376 0 0) 0x7f562813daf0 con 0x7f56280aeca0 2024-03-29T11:21:31.326 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:31.327 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:31.327 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:31.327 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:21:31.327 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 1 -- 172.21.15.170:0/4262698296 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5614003520 con 0x7f56280aeca0 2024-03-29T11:21:31.327 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 1 -- 172.21.15.170:0/4262698296 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919151477 0 0) 0x7f56280ae180 con 0x7f56280b0500 2024-03-29T11:21:31.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:31.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:31.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:31.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:21:31.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 1 -- 172.21.15.170:0/4262698296 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f562813daf0 con 0x7f56280b0500 2024-03-29T11:21:31.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 1 -- 172.21.15.170:0/4262698296 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1226756681 0 0) 0x7f5614003520 con 0x7f56280aeca0 2024-03-29T11:21:31.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 1 -- 172.21.15.170:0/4262698296 >> v1:172.21.15.170:6790/0 conn(0x7f56280b0500 legacy=0x7f562813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:31.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 1 -- 172.21.15.170:0/4262698296 >> v1:172.21.15.170:6789/0 conn(0x7f56280b08e0 legacy=0x7f562813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:31.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:31.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 1 monclient: found mon.noname-c 2024-03-29T11:21:31.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:31.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:21:31.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 1 -- 172.21.15.170:0/4262698296 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56280accf0 con 0x7f56280aeca0 2024-03-29T11:21:31.328 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 10 monclient: _finish_auth 0 2024-03-29T11:21:31.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:31.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.347+0000 7f561ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:31.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.348+0000 7f561ffff640 1 -- 172.21.15.170:0/4262698296 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5620001a10 con 0x7f56280aeca0 2024-03-29T11:21:31.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.348+0000 7f561ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:31.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.348+0000 7f561ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:31.329 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.348+0000 7f561ffff640 10 monclient: dump: 2024-03-29T11:21:31.329 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:31.329 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:31.329 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:31.329 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:31.329 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:31.329 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:31.329 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:31.329 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:31.330 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:31.330 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:31.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.348+0000 7f561ffff640 1 -- 172.21.15.170:0/4262698296 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5620001c30 con 0x7f56280aeca0 2024-03-29T11:21:31.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.348+0000 7f561ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:31.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.348+0000 7f561ffff640 1 -- 172.21.15.170:0/4262698296 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5620001fc0 con 0x7f56280aeca0 2024-03-29T11:21:31.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.348+0000 7f561ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:31.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.348+0000 7f561ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:31.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.348+0000 7f561ffff640 10 monclient: dump: 2024-03-29T11:21:31.330 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:31.330 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:31.330 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:31.330 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:31.330 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:31.331 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:31.331 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:31.331 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:31.331 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:31.331 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:31.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.348+0000 7f562f3fb640 5 monclient: authenticate success, global_id 4543 2024-03-29T11:21:31.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.348+0000 7f562f3fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:31.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.348+0000 7f562f3fb640 4 set_mon_vals no callback set 2024-03-29T11:21:31.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.349+0000 7f562f3fb640 10 monclient: shutdown 2024-03-29T11:21:31.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.349+0000 7f562f3fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:31.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.349+0000 7f562f3fb640 1 -- 172.21.15.170:0/4262698296 >> v1:172.21.15.184:6789/0 conn(0x7f56280aeca0 legacy=0x7f562814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:31.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.349+0000 7f562f3fb640 1 -- 172.21.15.170:0/4262698296 shutdown_connections 2024-03-29T11:21:31.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.349+0000 7f562f3fb640 1 -- 172.21.15.170:0/4262698296 >> 172.21.15.170:0/4262698296 conn(0x7f562806b660 msgr2=0x7f562806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:31.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.350+0000 7f562f3fb640 1 -- 172.21.15.170:0/4262698296 shutdown_connections 2024-03-29T11:21:31.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.350+0000 7f562f3fb640 1 -- 172.21.15.170:0/4262698296 wait complete. 2024-03-29T11:21:31.335 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.351+0000 7f562f3fb640 10 monclient: build_initial_monmap 2024-03-29T11:21:31.335 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.351+0000 7f562f3fb640 10 monclient: monmap: 2024-03-29T11:21:31.335 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:21:31.335 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:31.335 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:21:31.336 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:21:31.336 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:31.336 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:31.336 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:21:31.336 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:21:31.336 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:21:31.336 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:31.336 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.352+0000 7f562f3fb640 1 librados: starting msgr at 2024-03-29T11:21:31.336 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.352+0000 7f562f3fb640 1 librados: starting objecter 2024-03-29T11:21:31.336 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.352+0000 7f562f3fb640 1 Processor -- start 2024-03-29T11:21:31.336 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.352+0000 7f562f3fb640 1 -- start start 2024-03-29T11:21:31.336 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.352+0000 7f562f3fb640 1 librados: setting wanted keys 2024-03-29T11:21:31.336 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.352+0000 7f562f3fb640 1 librados: calling monclient init 2024-03-29T11:21:31.336 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.352+0000 7f562f3fb640 10 monclient: init 2024-03-29T11:21:31.337 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.353+0000 7f562f3fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:31.337 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.353+0000 7f562f3fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:31.337 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.353+0000 7f562f3fb640 10 monclient(hunting): picked mon.noname-a con 0x7f56280aeca0 addr v1:172.21.15.170:6789/0 2024-03-29T11:21:31.337 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.353+0000 7f562f3fb640 10 monclient(hunting): picked mon.noname-b con 0x7f56280b0500 addr v1:172.21.15.184:6789/0 2024-03-29T11:21:31.337 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.353+0000 7f562c996640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f56280b0500 0x7f56281cac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:44066/0 (socket says 172.21.15.170:44066) 2024-03-29T11:21:31.337 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.353+0000 7f562c996640 1 -- 172.21.15.170:0/75632839 learned_addr learned my addr 172.21.15.170:0/75632839 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:21:31.337 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.353+0000 7f562f3fb640 10 monclient(hunting): picked mon.noname-c con 0x7f562813bbf0 addr v1:172.21.15.170:6790/0 2024-03-29T11:21:31.337 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.353+0000 7f562f3fb640 1 -- 172.21.15.170:0/75632839 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56281cc160 con 0x7f56280aeca0 2024-03-29T11:21:31.337 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.353+0000 7f562f3fb640 1 -- 172.21.15.170:0/75632839 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56281ccd10 con 0x7f56280b0500 2024-03-29T11:21:31.337 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f562f3fb640 1 -- 172.21.15.170:0/75632839 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56281d8ab0 con 0x7f562813bbf0 2024-03-29T11:21:31.337 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 10 client.?.objecter ms_handle_connect 0x7f56280b0500 2024-03-29T11:21:31.337 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:31.337 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f562f3fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:31.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f562f3fb640 10 monclient(hunting): authenticate will time out at 1472.665142s 2024-03-29T11:21:31.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 10 client.?.objecter ms_handle_connect 0x7f56280aeca0 2024-03-29T11:21:31.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:31.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342532176 0 0) 0x7f56281ccd10 con 0x7f56280b0500 2024-03-29T11:21:31.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:31.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:31.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:31.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:21:31.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f560c004eb0 con 0x7f56280b0500 2024-03-29T11:21:31.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 10 client.?.objecter ms_handle_connect 0x7f562813bbf0 2024-03-29T11:21:31.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:31.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549094404 0 0) 0x7f56281d8ab0 con 0x7f562813bbf0 2024-03-29T11:21:31.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:31.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:31.339 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:31.339 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:21:31.339 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56281ccd10 con 0x7f562813bbf0 2024-03-29T11:21:31.339 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1835618437 0 0) 0x7f560c004eb0 con 0x7f56280b0500 2024-03-29T11:21:31.339 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56281d8ab0 con 0x7f56280b0500 2024-03-29T11:21:31.339 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.354+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f56240018d0 con 0x7f56280b0500 2024-03-29T11:21:31.339 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:31.339 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:31.339 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 10 monclient(hunting): dump: 2024-03-29T11:21:31.339 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:31.339 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:31.339 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:31.339 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:31.339 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:31.340 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:31.340 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:31.340 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:31.340 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:31.340 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:31.340 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:31.340 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810243792 0 0) 0x7f56281cc160 con 0x7f56280aeca0 2024-03-29T11:21:31.340 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:31.340 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:31.340 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:31.340 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:21:31.340 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f560c004eb0 con 0x7f56280aeca0 2024-03-29T11:21:31.340 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3664249246 0 0) 0x7f56281ccd10 con 0x7f562813bbf0 2024-03-29T11:21:31.340 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56281cc160 con 0x7f562813bbf0 2024-03-29T11:21:31.340 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5620001950 con 0x7f562813bbf0 2024-03-29T11:21:31.341 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:31.341 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:31.341 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 10 monclient(hunting): dump: 2024-03-29T11:21:31.341 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:31.341 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:31.341 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:31.341 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:31.341 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:31.341 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:31.341 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:31.341 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:31.341 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:31.341 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:31.341 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4066173290 0 0) 0x7f56281d8ab0 con 0x7f56280b0500 2024-03-29T11:21:31.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 >> v1:172.21.15.170:6790/0 conn(0x7f562813bbf0 legacy=0x7f56281cb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:31.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 >> v1:172.21.15.170:6789/0 conn(0x7f56280aeca0 legacy=0x7f56281c8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:31.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:31.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 1 monclient: found mon.b 2024-03-29T11:21:31.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:31.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:21:31.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56281d8c40 con 0x7f56280b0500 2024-03-29T11:21:31.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:31.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:31.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:31.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f562f3fb640 5 monclient: authenticate success, global_id 4546 2024-03-29T11:21:31.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f562f3fb640 10 monclient: _renew_subs 2024-03-29T11:21:31.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f562f3fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:21:31.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f562f3fb640 1 -- 172.21.15.170:0/75632839 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56281cba90 con 0x7f56280b0500 2024-03-29T11:21:31.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5624003610 con 0x7f56280b0500 2024-03-29T11:21:31.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:31.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f5624003960 con 0x7f56280b0500 2024-03-29T11:21:31.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:31.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:31.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561dffb640 10 monclient: dump: 2024-03-29T11:21:31.343 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:31.343 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:31.343 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:31.343 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:31.344 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:31.344 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:31.344 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:31.344 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:31.344 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:31.344 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:31.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f562f3fb640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:31.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f562f3fb640 10 monclient: _renew_subs 2024-03-29T11:21:31.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f562f3fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:21:31.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f562f3fb640 1 -- 172.21.15.170:0/75632839 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56281d9230 con 0x7f56280b0500 2024-03-29T11:21:31.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f561f7fe640 4 set_mon_vals no callback set 2024-03-29T11:21:31.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.355+0000 7f562f3fb640 1 librados: init done 2024-03-29T11:21:31.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.357+0000 7f562f3fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:31.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.357+0000 7f562f3fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:31.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.357+0000 7f562f3fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:21:31.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.357+0000 7f562f3fb640 1 -- 172.21.15.170:0/75632839 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56280ad0c0 con 0x7f56280b0500 2024-03-29T11:21:31.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.358+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f5624003ac0 con 0x7f56280b0500 2024-03-29T11:21:31.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.359+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f56240848d0 con 0x7f56280b0500 2024-03-29T11:21:31.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.359+0000 7f561dffb640 10 client.4546.objecter ms_dispatch 0x7f5628000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:31.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.359+0000 7f561dffb640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:31.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.359+0000 7f561dffb640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:31.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.359+0000 7f561dffb640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-29T11:21:31.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.359+0000 7f562d197640 10 monclient: get_auth_request con 0x7f560c06df20 auth_method 0 2024-03-29T11:21:31.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.359+0000 7f561dffb640 10 client.4546.objecter ms_handle_connect 0x7f560c06df20 2024-03-29T11:21:31.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.365+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f56240501b0 con 0x7f56280b0500 2024-03-29T11:21:31.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.365+0000 7f561dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:31.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.365+0000 7f561dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:31.725 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.749+0000 7f562f3fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:31.725 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.749+0000 7f562f3fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:31.725 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.749+0000 7f562f3fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:21:31.725 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.749+0000 7f562f3fb640 1 -- 172.21.15.170:0/75632839 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5628075f10 con 0x7f56280b0500 2024-03-29T11:21:31.726 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.750+0000 7f561dffb640 1 -- 172.21.15.170:0/75632839 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f56240520e0 con 0x7f56280b0500 2024-03-29T11:21:31.726 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.750+0000 7f561dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:31.726 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.750+0000 7f561dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:31.726 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:21:31.726 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:31.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.754+0000 7f562f3fb640 10 librados: watch_flush enter 2024-03-29T11:21:31.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.754+0000 7f562f3fb640 10 librados: watch_flush exit 2024-03-29T11:21:31.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.754+0000 7f562f3fb640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-29T11:21:31.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.754+0000 7f562f3fb640 10 client.4546.objecter successfully canceled tick 2024-03-29T11:21:31.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.754+0000 7f562f3fb640 1 -- 172.21.15.170:0/75632839 >> v1:172.21.15.170:6816/3298994444 conn(0x7f560c06df20 legacy=0x7f560c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:31.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.755+0000 7f562f3fb640 10 monclient: shutdown 2024-03-29T11:21:31.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.755+0000 7f562f3fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:31.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.755+0000 7f562f3fb640 1 -- 172.21.15.170:0/75632839 >> v1:172.21.15.184:6789/0 conn(0x7f56280b0500 legacy=0x7f56281cac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:31.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.755+0000 7f562f3fb640 1 -- 172.21.15.170:0/75632839 shutdown_connections 2024-03-29T11:21:31.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.755+0000 7f562f3fb640 1 -- 172.21.15.170:0/75632839 >> 172.21.15.170:0/75632839 conn(0x7f562806b660 msgr2=0x7f56280a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:31.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.755+0000 7f562f3fb640 1 -- 172.21.15.170:0/75632839 shutdown_connections 2024-03-29T11:21:31.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.755+0000 7f562f3fb640 1 -- 172.21.15.170:0/75632839 wait complete. 2024-03-29T11:21:31.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:31.755+0000 7f562f3fb640 1 librados: shutdown 2024-03-29T11:21:31.748 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:38.554 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:38.705 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.728+0000 7fdfa869a640 10 monclient: get_monmap_and_config 2024-03-29T11:21:38.706 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.728+0000 7fdfa869a640 10 monclient: build_initial_monmap 2024-03-29T11:21:38.706 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.728+0000 7fdfa869a640 10 monclient: monmap: 2024-03-29T11:21:38.706 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:21:38.706 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:38.706 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:21:38.730367+0000 2024-03-29T11:21:38.706 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:21:38.730367+0000 2024-03-29T11:21:38.706 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:38.706 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:38.706 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:21:38.706 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:21:38.706 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:21:38.706 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:38.707 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.729+0000 7fdfa869a640 1 Processor -- start 2024-03-29T11:21:38.707 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.729+0000 7fdfa869a640 1 -- start start 2024-03-29T11:21:38.707 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.729+0000 7fdfa869a640 10 monclient: init 2024-03-29T11:21:38.709 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.730+0000 7fdfa869a640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:38.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.730+0000 7fdfa869a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:38.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.730+0000 7fdfa869a640 10 monclient(hunting): picked mon.noname-b con 0x7fdfa006d260 addr v1:172.21.15.170:6790/0 2024-03-29T11:21:38.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.730+0000 7fdfa869a640 10 monclient(hunting): picked mon.noname-a con 0x7fdfa00723b0 addr v1:172.21.15.170:6789/0 2024-03-29T11:21:38.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.730+0000 7fdfa869a640 10 monclient(hunting): picked mon.noname-c con 0x7fdfa0146b80 addr v1:172.21.15.184:6789/0 2024-03-29T11:21:38.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.730+0000 7fdfa869a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa00701d0 con 0x7fdfa00723b0 2024-03-29T11:21:38.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.730+0000 7fdfa869a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa0070440 con 0x7fdfa0146b80 2024-03-29T11:21:38.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.731+0000 7fdfa869a640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa006c2c0 con 0x7fdfa006d260 2024-03-29T11:21:38.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.731+0000 7fdfa869a640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:38.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.731+0000 7fdfa869a640 10 monclient(hunting): authenticate will time out at 1480.043032s 2024-03-29T11:21:38.717 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.740+0000 7fdfa5c35640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fdfa00723b0 0x7fdfa0146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:51211/0 (socket says 172.21.15.170:51211) 2024-03-29T11:21:38.717 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.740+0000 7fdfa5c35640 1 -- 172.21.15.170:0/115199130 learned_addr learned my addr 172.21.15.170:0/115199130 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:21:38.723 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.746+0000 7fdfa5434640 1 -- 172.21.15.170:0/115199130 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268052620 0 0) 0x7fdfa0070440 con 0x7fdfa0146b80 2024-03-29T11:21:38.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.746+0000 7fdfa5434640 10 monclient(hunting): _negotiate 2024-03-29T11:21:38.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.746+0000 7fdfa5434640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:38.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.746+0000 7fdfa5434640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:38.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.746+0000 7fdfa5434640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:21:38.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.746+0000 7fdfa5434640 1 -- 172.21.15.170:0/115199130 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf8c003520 con 0x7fdfa0146b80 2024-03-29T11:21:38.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.746+0000 7fdfa5434640 1 -- 172.21.15.170:0/115199130 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651543796 0 0) 0x7fdfa00701d0 con 0x7fdfa00723b0 2024-03-29T11:21:38.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.746+0000 7fdfa5434640 10 monclient(hunting): _negotiate 2024-03-29T11:21:38.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.746+0000 7fdfa5434640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:38.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.746+0000 7fdfa5434640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:38.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.746+0000 7fdfa5434640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:21:38.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.746+0000 7fdfa5434640 1 -- 172.21.15.170:0/115199130 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf8c003840 con 0x7fdfa00723b0 2024-03-29T11:21:38.729 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.746+0000 7fdfa5434640 1 -- 172.21.15.170:0/115199130 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2971370052 0 0) 0x7fdf8c003840 con 0x7fdfa00723b0 2024-03-29T11:21:38.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.746+0000 7fdfa5434640 1 -- 172.21.15.170:0/115199130 >> v1:172.21.15.170:6790/0 conn(0x7fdfa006d260 legacy=0x7fdfa0071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:38.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.746+0000 7fdfa5434640 1 -- 172.21.15.170:0/115199130 >> v1:172.21.15.184:6789/0 conn(0x7fdfa0146b80 legacy=0x7fdfa0151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:38.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.746+0000 7fdfa5434640 10 monclient: _finish_hunting 0 2024-03-29T11:21:38.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.746+0000 7fdfa5434640 1 monclient: found mon.noname-a 2024-03-29T11:21:38.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.746+0000 7fdfa5434640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:38.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.747+0000 7fdfa5434640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:21:38.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.747+0000 7fdfa5434640 1 -- 172.21.15.170:0/115199130 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa006c7e0 con 0x7fdfa00723b0 2024-03-29T11:21:38.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.747+0000 7fdfa5434640 10 monclient: _finish_auth 0 2024-03-29T11:21:38.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.747+0000 7fdfa5434640 10 monclient: _check_auth_tickets 2024-03-29T11:21:38.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.747+0000 7fdfa5434640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:38.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.747+0000 7fdfa869a640 5 monclient: authenticate success, global_id 4563 2024-03-29T11:21:38.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.747+0000 7fdfa869a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:38.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.747+0000 7fdfa5434640 1 -- 172.21.15.170:0/115199130 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fdf98001970 con 0x7fdfa00723b0 2024-03-29T11:21:38.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.747+0000 7fdfa5434640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:38.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.747+0000 7fdfa5434640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:38.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.747+0000 7fdfa5434640 10 monclient: dump: 2024-03-29T11:21:38.731 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:38.731 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:38.731 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:38.731 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:38.731 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:38.731 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:38.731 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:38.731 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:38.731 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:38.731 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:38.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.747+0000 7fdfa869a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:38.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.747+0000 7fdfa5434640 1 -- 172.21.15.170:0/115199130 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf98001b50 con 0x7fdfa00723b0 2024-03-29T11:21:38.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.747+0000 7fdfa5434640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:38.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.747+0000 7fdfa869a640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:38.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.747+0000 7fdfa869a640 4 set_mon_vals no callback set 2024-03-29T11:21:38.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.748+0000 7fdfa869a640 10 monclient: shutdown 2024-03-29T11:21:38.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.748+0000 7fdfa869a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:38.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.748+0000 7fdfa869a640 1 -- 172.21.15.170:0/115199130 >> v1:172.21.15.170:6789/0 conn(0x7fdfa00723b0 legacy=0x7fdfa0146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:38.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.748+0000 7fdfa5434640 1 -- 172.21.15.170:0/115199130 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fdf98001970 con 0x7fdfa00723b0 2024-03-29T11:21:38.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.748+0000 7fdfa5434640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:21:38.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.749+0000 7fdfa869a640 1 -- 172.21.15.170:0/115199130 shutdown_connections 2024-03-29T11:21:38.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.749+0000 7fdfa869a640 1 -- 172.21.15.170:0/115199130 >> 172.21.15.170:0/115199130 conn(0x7fdfa0067500 msgr2=0x7fdfa00b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:38.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.750+0000 7fdfa869a640 1 -- 172.21.15.170:0/115199130 shutdown_connections 2024-03-29T11:21:38.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.750+0000 7fdfa869a640 1 -- 172.21.15.170:0/115199130 wait complete. 2024-03-29T11:21:38.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.751+0000 7fdfa869a640 10 monclient: build_initial_monmap 2024-03-29T11:21:38.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.751+0000 7fdfa869a640 10 monclient: monmap: 2024-03-29T11:21:38.733 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:21:38.733 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:38.733 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:21:38.733 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:21:38.733 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:38.733 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:38.733 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:21:38.733 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:21:38.734 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:21:38.734 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:38.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.752+0000 7fdfa869a640 1 librados: starting msgr at 2024-03-29T11:21:38.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.752+0000 7fdfa869a640 1 librados: starting objecter 2024-03-29T11:21:38.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.752+0000 7fdfa869a640 1 Processor -- start 2024-03-29T11:21:38.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.752+0000 7fdfa869a640 1 -- start start 2024-03-29T11:21:38.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.752+0000 7fdfa869a640 1 librados: setting wanted keys 2024-03-29T11:21:38.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.752+0000 7fdfa869a640 1 librados: calling monclient init 2024-03-29T11:21:38.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.752+0000 7fdfa869a640 10 monclient: init 2024-03-29T11:21:38.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.753+0000 7fdfa869a640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:38.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.753+0000 7fdfa869a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:38.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.753+0000 7fdfa869a640 10 monclient(hunting): picked mon.noname-a con 0x7fdfa006d260 addr v1:172.21.15.170:6789/0 2024-03-29T11:21:38.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.753+0000 7fdfa869a640 10 monclient(hunting): picked mon.noname-c con 0x7fdfa00723b0 addr v1:172.21.15.170:6790/0 2024-03-29T11:21:38.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.753+0000 7fdfa869a640 10 monclient(hunting): picked mon.noname-b con 0x7fdfa0146b80 addr v1:172.21.15.184:6789/0 2024-03-29T11:21:38.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.753+0000 7fdfa869a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa00adae0 con 0x7fdfa006d260 2024-03-29T11:21:38.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.753+0000 7fdfa869a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa00addc0 con 0x7fdfa0146b80 2024-03-29T11:21:38.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.753+0000 7fdfa869a640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa00ae0a0 con 0x7fdfa00723b0 2024-03-29T11:21:38.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.753+0000 7fdfa869a640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:38.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.753+0000 7fdfa869a640 10 monclient(hunting): authenticate will time out at 1480.065365s 2024-03-29T11:21:38.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.757+0000 7fdfa6436640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fdfa006d260 0x7fdfa00ac5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:51345/0 (socket says 172.21.15.170:51345) 2024-03-29T11:21:38.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.757+0000 7fdfa6436640 1 -- 172.21.15.170:0/347383427 learned_addr learned my addr 172.21.15.170:0/347383427 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:21:38.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.760+0000 7fdf96ffd640 10 client.?.objecter ms_handle_connect 0x7fdfa00723b0 2024-03-29T11:21:38.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.760+0000 7fdf96ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:38.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.760+0000 7fdf96ffd640 1 -- 172.21.15.170:0/347383427 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704435377 0 0) 0x7fdfa00ae0a0 con 0x7fdfa00723b0 2024-03-29T11:21:38.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.760+0000 7fdf96ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:38.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.760+0000 7fdf96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:38.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.760+0000 7fdf96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:38.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.760+0000 7fdf96ffd640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:21:38.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.760+0000 7fdf96ffd640 1 -- 172.21.15.170:0/347383427 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf84004eb0 con 0x7fdfa00723b0 2024-03-29T11:21:38.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.761+0000 7fdf96ffd640 10 client.?.objecter ms_handle_connect 0x7fdfa006d260 2024-03-29T11:21:38.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.761+0000 7fdf96ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:38.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.761+0000 7fdf96ffd640 10 client.?.objecter ms_handle_connect 0x7fdfa0146b80 2024-03-29T11:21:38.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.761+0000 7fdf96ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:38.740 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.761+0000 7fdf96ffd640 1 -- 172.21.15.170:0/347383427 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1787128882 0 0) 0x7fdf84004eb0 con 0x7fdfa00723b0 2024-03-29T11:21:38.740 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.761+0000 7fdf96ffd640 1 -- 172.21.15.170:0/347383427 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfa00ae0a0 con 0x7fdfa00723b0 2024-03-29T11:21:38.740 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.761+0000 7fdf96ffd640 1 -- 172.21.15.170:0/347383427 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fdf980020c0 con 0x7fdfa00723b0 2024-03-29T11:21:38.740 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.761+0000 7fdf96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:38.740 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.761+0000 7fdf96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:38.740 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.761+0000 7fdf96ffd640 10 monclient(hunting): dump: 2024-03-29T11:21:38.740 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:38.740 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:38.740 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:38.740 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:38.740 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:38.740 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:38.740 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:38.740 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:38.740 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:38.741 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:38.741 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.761+0000 7fdf96ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:38.742 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.762+0000 7fdf96ffd640 1 -- 172.21.15.170:0/347383427 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821016612 0 0) 0x7fdfa00addc0 con 0x7fdfa0146b80 2024-03-29T11:21:38.742 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.762+0000 7fdf96ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:38.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.762+0000 7fdf96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:38.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.762+0000 7fdf96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:38.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.762+0000 7fdf96ffd640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:21:38.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.762+0000 7fdf96ffd640 1 -- 172.21.15.170:0/347383427 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf84004eb0 con 0x7fdfa0146b80 2024-03-29T11:21:38.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.762+0000 7fdf96ffd640 1 -- 172.21.15.170:0/347383427 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091123965 0 0) 0x7fdfa00adae0 con 0x7fdfa006d260 2024-03-29T11:21:38.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.762+0000 7fdf96ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:38.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.762+0000 7fdf96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:38.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.762+0000 7fdf96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:38.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.762+0000 7fdf96ffd640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:21:38.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.762+0000 7fdf96ffd640 1 -- 172.21.15.170:0/347383427 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfa00addc0 con 0x7fdfa006d260 2024-03-29T11:21:38.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.766+0000 7fdf96ffd640 1 -- 172.21.15.170:0/347383427 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1219896256 0 0) 0x7fdfa00ae0a0 con 0x7fdfa00723b0 2024-03-29T11:21:38.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.766+0000 7fdf96ffd640 1 -- 172.21.15.170:0/347383427 >> v1:172.21.15.184:6789/0 conn(0x7fdfa0146b80 legacy=0x7fdfa01ffb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:38.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.766+0000 7fdf96ffd640 1 -- 172.21.15.170:0/347383427 >> v1:172.21.15.170:6789/0 conn(0x7fdfa006d260 legacy=0x7fdfa00ac5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:38.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.766+0000 7fdf96ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:38.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.766+0000 7fdf96ffd640 1 monclient: found mon.c 2024-03-29T11:21:38.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.766+0000 7fdf96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:38.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.766+0000 7fdf96ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:38.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.766+0000 7fdf96ffd640 1 -- 172.21.15.170:0/347383427 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa00aba40 con 0x7fdfa00723b0 2024-03-29T11:21:38.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.766+0000 7fdf96ffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:38.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.766+0000 7fdf96ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:38.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.766+0000 7fdf96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:38.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.768+0000 7fdfa869a640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:21:38.747 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.768+0000 7fdfa869a640 10 monclient: _renew_subs 2024-03-29T11:21:38.747 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.768+0000 7fdfa869a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:38.747 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.768+0000 7fdfa869a640 1 -- 172.21.15.170:0/347383427 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa00af4d0 con 0x7fdfa00723b0 2024-03-29T11:21:38.747 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.768+0000 7fdfa869a640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:38.747 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.768+0000 7fdfa869a640 10 monclient: _renew_subs 2024-03-29T11:21:38.747 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.768+0000 7fdfa869a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:38.747 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.768+0000 7fdfa869a640 1 -- 172.21.15.170:0/347383427 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa00a9dd0 con 0x7fdfa00723b0 2024-03-29T11:21:38.747 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.768+0000 7fdfa869a640 1 librados: init done 2024-03-29T11:21:38.748 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.770+0000 7fdf96ffd640 1 -- 172.21.15.170:0/347383427 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf98004e10 con 0x7fdfa00723b0 2024-03-29T11:21:38.748 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.770+0000 7fdf96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:38.748 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.770+0000 7fdf96ffd640 1 -- 172.21.15.170:0/347383427 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fdf98005120 con 0x7fdfa00723b0 2024-03-29T11:21:38.748 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.770+0000 7fdf96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:38.748 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.770+0000 7fdf96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:38.748 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.770+0000 7fdf96ffd640 10 monclient: dump: 2024-03-29T11:21:38.748 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:38.748 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:38.748 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:38.748 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:38.748 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:38.748 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:38.748 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:38.749 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:38.749 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:38.749 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:38.749 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.770+0000 7fdfa4c33640 4 set_mon_vals no callback set 2024-03-29T11:21:38.749 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.771+0000 7fdfa869a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:38.749 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.771+0000 7fdfa869a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:38.749 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.771+0000 7fdfa869a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:38.749 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.771+0000 7fdfa869a640 1 -- 172.21.15.170:0/347383427 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf88005080 con 0x7fdfa00723b0 2024-03-29T11:21:38.751 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.773+0000 7fdf96ffd640 1 -- 172.21.15.170:0/347383427 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fdf9801b0f0 con 0x7fdfa00723b0 2024-03-29T11:21:38.753 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.774+0000 7fdf96ffd640 1 -- 172.21.15.170:0/347383427 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fdf98084ee0 con 0x7fdfa00723b0 2024-03-29T11:21:38.753 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.774+0000 7fdf96ffd640 10 client.4631.objecter ms_dispatch 0x7fdfa0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:38.753 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.774+0000 7fdf96ffd640 3 client.4631.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:38.753 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.774+0000 7fdf96ffd640 3 client.4631.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:38.753 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.774+0000 7fdf96ffd640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-29T11:21:38.754 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.775+0000 7fdfa6436640 10 monclient: get_auth_request con 0x7fdf8406ded0 auth_method 0 2024-03-29T11:21:38.754 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.775+0000 7fdf96ffd640 10 client.4631.objecter ms_handle_connect 0x7fdf8406ded0 2024-03-29T11:21:38.764 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.785+0000 7fdf96ffd640 1 -- 172.21.15.170:0/347383427 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdf980507c0 con 0x7fdfa00723b0 2024-03-29T11:21:38.764 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.785+0000 7fdf96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:38.764 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:38.785+0000 7fdf96ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:39.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.137+0000 7fdfa869a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:39.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.137+0000 7fdfa869a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:39.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.137+0000 7fdfa869a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:21:39.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.137+0000 7fdfa869a640 1 -- 172.21.15.170:0/347383427 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf88005330 con 0x7fdfa00723b0 2024-03-29T11:21:39.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.139+0000 7fdf96ffd640 1 -- 172.21.15.170:0/347383427 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fdf980526f0 con 0x7fdfa00723b0 2024-03-29T11:21:39.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.139+0000 7fdf96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:39.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.139+0000 7fdf96ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:39.116 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:21:39.116 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:39.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.142+0000 7fdfa869a640 10 librados: watch_flush enter 2024-03-29T11:21:39.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.142+0000 7fdfa869a640 10 librados: watch_flush exit 2024-03-29T11:21:39.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.142+0000 7fdfa869a640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-29T11:21:39.119 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.142+0000 7fdfa869a640 10 client.4631.objecter successfully canceled tick 2024-03-29T11:21:39.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.145+0000 7fdfa869a640 1 -- 172.21.15.170:0/347383427 >> v1:172.21.15.170:6816/3298994444 conn(0x7fdf8406ded0 legacy=0x7fdf84070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:39.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.145+0000 7fdfa869a640 10 monclient: shutdown 2024-03-29T11:21:39.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.145+0000 7fdfa869a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:39.122 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.145+0000 7fdfa869a640 1 -- 172.21.15.170:0/347383427 >> v1:172.21.15.170:6790/0 conn(0x7fdfa00723b0 legacy=0x7fdfa00a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:39.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.146+0000 7fdfa869a640 1 -- 172.21.15.170:0/347383427 shutdown_connections 2024-03-29T11:21:39.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.146+0000 7fdfa869a640 1 -- 172.21.15.170:0/347383427 >> 172.21.15.170:0/347383427 conn(0x7fdfa0067500 msgr2=0x7fdfa00707b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:39.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.147+0000 7fdfa869a640 1 -- 172.21.15.170:0/347383427 shutdown_connections 2024-03-29T11:21:39.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.147+0000 7fdfa869a640 1 -- 172.21.15.170:0/347383427 wait complete. 2024-03-29T11:21:39.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:39.147+0000 7fdfa869a640 1 librados: shutdown 2024-03-29T11:21:39.147 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:45.954 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:46.101 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.124+0000 7f34a0520640 10 monclient: get_monmap_and_config 2024-03-29T11:21:46.101 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.124+0000 7f34a0520640 10 monclient: build_initial_monmap 2024-03-29T11:21:46.101 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.124+0000 7f34a0520640 10 monclient: monmap: 2024-03-29T11:21:46.101 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:21:46.102 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:46.102 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:21:46.126020+0000 2024-03-29T11:21:46.102 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:21:46.126020+0000 2024-03-29T11:21:46.102 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:46.102 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:46.102 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:21:46.102 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:21:46.102 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:21:46.102 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:46.104 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.127+0000 7f34a0520640 1 Processor -- start 2024-03-29T11:21:46.105 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.128+0000 7f34a0520640 1 -- start start 2024-03-29T11:21:46.105 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.128+0000 7f34a0520640 10 monclient: init 2024-03-29T11:21:46.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.129+0000 7f34a0520640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:46.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.129+0000 7f34a0520640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:46.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.129+0000 7f34a0520640 10 monclient(hunting): picked mon.noname-a con 0x7f34980b0770 addr v1:172.21.15.170:6789/0 2024-03-29T11:21:46.106 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.129+0000 7f34a0520640 10 monclient(hunting): picked mon.noname-c con 0x7f349814a700 addr v1:172.21.15.184:6789/0 2024-03-29T11:21:46.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.130+0000 7f349eabd640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f34980ac5a0 0x7f3498151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:43900/0 (socket says 172.21.15.170:43900) 2024-03-29T11:21:46.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.130+0000 7f349eabd640 1 -- 172.21.15.170:0/4202381544 learned_addr learned my addr 172.21.15.170:0/4202381544 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:21:46.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.131+0000 7f34a0520640 10 monclient(hunting): picked mon.noname-b con 0x7f34980ac5a0 addr v1:172.21.15.170:6790/0 2024-03-29T11:21:46.107 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.131+0000 7f34a0520640 1 -- 172.21.15.170:0/4202381544 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34980aba70 con 0x7f34980b0770 2024-03-29T11:21:46.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.132+0000 7f34a0520640 1 -- 172.21.15.170:0/4202381544 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34980aecb0 con 0x7f349814a700 2024-03-29T11:21:46.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.132+0000 7f34a0520640 1 -- 172.21.15.170:0/4202381544 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34980af150 con 0x7f34980ac5a0 2024-03-29T11:21:46.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.132+0000 7f34a0520640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:46.109 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.132+0000 7f34a0520640 10 monclient(hunting): authenticate will time out at 1487.444548s 2024-03-29T11:21:46.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.132+0000 7f349d2ba640 1 -- 172.21.15.170:0/4202381544 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3530070487 0 0) 0x7f34980af150 con 0x7f34980ac5a0 2024-03-29T11:21:46.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.132+0000 7f349d2ba640 10 monclient(hunting): _negotiate 2024-03-29T11:21:46.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.132+0000 7f349d2ba640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:46.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.132+0000 7f349d2ba640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:46.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.132+0000 7f349d2ba640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:21:46.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.132+0000 7f349d2ba640 1 -- 172.21.15.170:0/4202381544 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3480003520 con 0x7f34980ac5a0 2024-03-29T11:21:46.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.133+0000 7f349d2ba640 1 -- 172.21.15.170:0/4202381544 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707159698 0 0) 0x7f34980aecb0 con 0x7f349814a700 2024-03-29T11:21:46.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.133+0000 7f349d2ba640 10 monclient(hunting): _negotiate 2024-03-29T11:21:46.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.133+0000 7f349d2ba640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:46.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.133+0000 7f349d2ba640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:46.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.133+0000 7f349d2ba640 10 monclient(hunting): my global_id is 4555 2024-03-29T11:21:46.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.133+0000 7f349d2ba640 1 -- 172.21.15.170:0/4202381544 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34980af150 con 0x7f349814a700 2024-03-29T11:21:46.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.133+0000 7f349d2ba640 1 -- 172.21.15.170:0/4202381544 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1866798854 0 0) 0x7f3480003520 con 0x7f34980ac5a0 2024-03-29T11:21:46.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.133+0000 7f349d2ba640 1 -- 172.21.15.170:0/4202381544 >> v1:172.21.15.184:6789/0 conn(0x7f349814a700 legacy=0x7f34980abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:46.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.133+0000 7f349d2ba640 1 -- 172.21.15.170:0/4202381544 >> v1:172.21.15.170:6789/0 conn(0x7f34980b0770 legacy=0x7f3498149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:46.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.134+0000 7f349d2ba640 10 monclient: _finish_hunting 0 2024-03-29T11:21:46.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.134+0000 7f349d2ba640 1 monclient: found mon.noname-b 2024-03-29T11:21:46.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.134+0000 7f349d2ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:46.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.134+0000 7f349d2ba640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:21:46.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.134+0000 7f349d2ba640 1 -- 172.21.15.170:0/4202381544 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34980ac9d0 con 0x7f34980ac5a0 2024-03-29T11:21:46.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.134+0000 7f349d2ba640 10 monclient: _finish_auth 0 2024-03-29T11:21:46.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.134+0000 7f349d2ba640 10 monclient: _check_auth_tickets 2024-03-29T11:21:46.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.134+0000 7f349d2ba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:46.115 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.134+0000 7f349d2ba640 1 -- 172.21.15.170:0/4202381544 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3490001690 con 0x7f34980ac5a0 2024-03-29T11:21:46.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.134+0000 7f349d2ba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:46.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.134+0000 7f349d2ba640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:46.116 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.134+0000 7f349d2ba640 10 monclient: dump: 2024-03-29T11:21:46.116 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:46.116 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:46.116 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:46.116 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:46.116 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:46.116 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:46.116 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:46.116 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:46.116 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:46.116 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:46.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.134+0000 7f34a0520640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:21:46.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.134+0000 7f34a0520640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:46.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.135+0000 7f349d2ba640 1 -- 172.21.15.170:0/4202381544 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34900017f0 con 0x7f34980ac5a0 2024-03-29T11:21:46.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.135+0000 7f349d2ba640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:46.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.135+0000 7f349d2ba640 1 -- 172.21.15.170:0/4202381544 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3490001c40 con 0x7f34980ac5a0 2024-03-29T11:21:46.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.135+0000 7f349d2ba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:46.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.135+0000 7f349d2ba640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:46.117 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.135+0000 7f349d2ba640 10 monclient: dump: 2024-03-29T11:21:46.117 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:46.117 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:46.117 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:46.117 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:46.117 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:46.117 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:46.117 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:46.118 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:46.118 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:46.118 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:46.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.135+0000 7f34a0520640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:46.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.135+0000 7f34a0520640 4 set_mon_vals no callback set 2024-03-29T11:21:46.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.136+0000 7f34a0520640 10 monclient: shutdown 2024-03-29T11:21:46.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.136+0000 7f34a0520640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:46.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.136+0000 7f34a0520640 1 -- 172.21.15.170:0/4202381544 >> v1:172.21.15.170:6790/0 conn(0x7f34980ac5a0 legacy=0x7f3498151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:46.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.136+0000 7f34a0520640 1 -- 172.21.15.170:0/4202381544 shutdown_connections 2024-03-29T11:21:46.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.136+0000 7f34a0520640 1 -- 172.21.15.170:0/4202381544 >> 172.21.15.170:0/4202381544 conn(0x7f349806b660 msgr2=0x7f349806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:46.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.137+0000 7f34a0520640 1 -- 172.21.15.170:0/4202381544 shutdown_connections 2024-03-29T11:21:46.118 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.137+0000 7f34a0520640 1 -- 172.21.15.170:0/4202381544 wait complete. 2024-03-29T11:21:46.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.137+0000 7f34a0520640 10 monclient: build_initial_monmap 2024-03-29T11:21:46.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.137+0000 7f34a0520640 10 monclient: monmap: 2024-03-29T11:21:46.123 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:21:46.123 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:46.123 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:21:46.123 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:21:46.123 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:46.123 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:46.123 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:21:46.123 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:21:46.123 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:21:46.123 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:46.123 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.138+0000 7f34a0520640 1 librados: starting msgr at 2024-03-29T11:21:46.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.138+0000 7f34a0520640 1 librados: starting objecter 2024-03-29T11:21:46.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.138+0000 7f34a0520640 1 Processor -- start 2024-03-29T11:21:46.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.138+0000 7f34a0520640 1 -- start start 2024-03-29T11:21:46.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.138+0000 7f34a0520640 1 librados: setting wanted keys 2024-03-29T11:21:46.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.138+0000 7f34a0520640 1 librados: calling monclient init 2024-03-29T11:21:46.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.138+0000 7f34a0520640 10 monclient: init 2024-03-29T11:21:46.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.139+0000 7f34a0520640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:46.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.139+0000 7f34a0520640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:46.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.139+0000 7f34a0520640 10 monclient(hunting): picked mon.noname-c con 0x7f34980ac5a0 addr v1:172.21.15.170:6790/0 2024-03-29T11:21:46.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.139+0000 7f34a0520640 10 monclient(hunting): picked mon.noname-b con 0x7f34980b0770 addr v1:172.21.15.184:6789/0 2024-03-29T11:21:46.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.140+0000 7f34a0520640 10 monclient(hunting): picked mon.noname-a con 0x7f349814a700 addr v1:172.21.15.170:6789/0 2024-03-29T11:21:46.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.140+0000 7f34a0520640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3498143f40 con 0x7f349814a700 2024-03-29T11:21:46.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.140+0000 7f34a0520640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f349813f910 con 0x7f34980b0770 2024-03-29T11:21:46.124 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.140+0000 7f34a0520640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f349813faa0 con 0x7f34980ac5a0 2024-03-29T11:21:46.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.140+0000 7f34a0520640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:46.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.140+0000 7f34a0520640 10 monclient(hunting): authenticate will time out at 1487.452867s 2024-03-29T11:21:46.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.143+0000 7f349eabd640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f349814a700 0x7f34981d28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:53453/0 (socket says 172.21.15.170:53453) 2024-03-29T11:21:46.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.143+0000 7f349eabd640 1 -- 172.21.15.170:0/1064590592 learned_addr learned my addr 172.21.15.170:0/1064590592 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:21:46.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.143+0000 7f349e2bc640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f34980ac5a0 0x7f3498143290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:44510/0 (socket says 172.21.15.170:44510) 2024-03-29T11:21:46.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.144+0000 7f3496ffd640 10 client.?.objecter ms_handle_connect 0x7f349814a700 2024-03-29T11:21:46.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.144+0000 7f3496ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:46.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.144+0000 7f3496ffd640 10 client.?.objecter ms_handle_connect 0x7f34980ac5a0 2024-03-29T11:21:46.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.144+0000 7f3496ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:46.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.144+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979330680 0 0) 0x7f3498143f40 con 0x7f349814a700 2024-03-29T11:21:46.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.144+0000 7f3496ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:46.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.144+0000 7f3496ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:46.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.144+0000 7f3496ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:46.125 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.144+0000 7f3496ffd640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:21:46.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.144+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3484004eb0 con 0x7f349814a700 2024-03-29T11:21:46.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.144+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411710290 0 0) 0x7f349813faa0 con 0x7f34980ac5a0 2024-03-29T11:21:46.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.144+0000 7f3496ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:46.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.144+0000 7f3496ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:46.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.144+0000 7f3496ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:46.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.144+0000 7f3496ffd640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:21:46.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.144+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3498143f40 con 0x7f34980ac5a0 2024-03-29T11:21:46.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.144+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 538261523 0 0) 0x7f3484004eb0 con 0x7f349814a700 2024-03-29T11:21:46.126 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.145+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f349813faa0 con 0x7f349814a700 2024-03-29T11:21:46.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3490001b00 con 0x7f349814a700 2024-03-29T11:21:46.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:46.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:46.130 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 10 monclient(hunting): dump: 2024-03-29T11:21:46.130 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:46.130 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:46.130 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:46.130 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:46.130 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:46.131 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:46.131 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:46.131 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:46.131 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:46.131 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:46.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:46.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4253590136 0 0) 0x7f3498143f40 con 0x7f34980ac5a0 2024-03-29T11:21:46.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3484004eb0 con 0x7f34980ac5a0 2024-03-29T11:21:46.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f348c0017e0 con 0x7f34980ac5a0 2024-03-29T11:21:46.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:46.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:46.131 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 10 monclient(hunting): dump: 2024-03-29T11:21:46.131 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:46.131 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:46.132 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:46.132 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:46.132 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:46.132 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:46.132 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:46.132 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:46.132 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:46.132 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:46.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 544588070 0 0) 0x7f349813faa0 con 0x7f349814a700 2024-03-29T11:21:46.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 >> v1:172.21.15.170:6790/0 conn(0x7f34980ac5a0 legacy=0x7f3498143290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:46.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 >> v1:172.21.15.184:6789/0 conn(0x7f34980b0770 legacy=0x7f34981d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:46.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:46.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 1 monclient: found mon.a 2024-03-29T11:21:46.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:46.132 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:21:46.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34981439a0 con 0x7f349814a700 2024-03-29T11:21:46.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:46.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:46.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f3496ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:46.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f34a0520640 5 monclient: authenticate success, global_id 4569 2024-03-29T11:21:46.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f34a0520640 10 monclient: _renew_subs 2024-03-29T11:21:46.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f34a0520640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:21:46.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.147+0000 7f34a0520640 1 -- 172.21.15.170:0/1064590592 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3498143c50 con 0x7f349814a700 2024-03-29T11:21:46.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.148+0000 7f34a0520640 10 client.4569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:46.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.148+0000 7f34a0520640 10 monclient: _renew_subs 2024-03-29T11:21:46.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.148+0000 7f34a0520640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:21:46.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.148+0000 7f34a0520640 1 -- 172.21.15.170:0/1064590592 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34981d9f20 con 0x7f349814a700 2024-03-29T11:21:46.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.148+0000 7f34a0520640 1 librados: init done 2024-03-29T11:21:46.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.150+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34900011f0 con 0x7f349814a700 2024-03-29T11:21:46.133 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.150+0000 7f3496ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:46.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.150+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f3490004d90 con 0x7f349814a700 2024-03-29T11:21:46.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.150+0000 7f3496ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:46.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.150+0000 7f3496ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:46.134 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.150+0000 7f3496ffd640 10 monclient: dump: 2024-03-29T11:21:46.134 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:46.134 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:46.134 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:46.134 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:46.134 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:46.134 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:46.134 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:46.134 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:46.134 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:46.134 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:46.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.150+0000 7f3494ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:46.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.150+0000 7f3494ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:46.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.150+0000 7f3494ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:21:46.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.150+0000 7f3494ff9640 1 -- 172.21.15.170:0/1064590592 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34980ab560 con 0x7f349814a700 2024-03-29T11:21:46.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.151+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f349001ad60 con 0x7f349814a700 2024-03-29T11:21:46.135 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.152+0000 7f349cab9640 4 set_mon_vals no callback set 2024-03-29T11:21:46.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.153+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7f3490084b30 con 0x7f349814a700 2024-03-29T11:21:46.136 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.153+0000 7f3496ffd640 10 client.4569.objecter ms_dispatch 0x7f3498000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:46.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.153+0000 7f3496ffd640 3 client.4569.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:46.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.153+0000 7f3496ffd640 3 client.4569.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:46.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.153+0000 7f3496ffd640 20 client.4569.objecter dump_active .. 0 homeless 2024-03-29T11:21:46.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.153+0000 7f349e2bc640 10 monclient: get_auth_request con 0x7f348406de80 auth_method 0 2024-03-29T11:21:46.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.154+0000 7f3496ffd640 10 client.4569.objecter ms_handle_connect 0x7f348406de80 2024-03-29T11:21:46.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.159+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3490050240 con 0x7f349814a700 2024-03-29T11:21:46.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.159+0000 7f3496ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:46.137 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.159+0000 7f3496ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:46.468 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.491+0000 7f3494ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:46.468 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.491+0000 7f3494ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:46.468 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.491+0000 7f3494ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:21:46.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.491+0000 7f3494ff9640 1 -- 172.21.15.170:0/1064590592 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34980afd20 con 0x7f349814a700 2024-03-29T11:21:46.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.492+0000 7f3496ffd640 1 -- 172.21.15.170:0/1064590592 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7f3490052340 con 0x7f349814a700 2024-03-29T11:21:46.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.492+0000 7f3496ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:46.469 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.492+0000 7f3496ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:46.469 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:21:46.470 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:46.473 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.496+0000 7f3494ff9640 10 librados: watch_flush enter 2024-03-29T11:21:46.473 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.496+0000 7f3494ff9640 10 librados: watch_flush exit 2024-03-29T11:21:46.473 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.496+0000 7f3494ff9640 20 client.4569.objecter shutdown clearing up homeless session... 2024-03-29T11:21:46.473 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.496+0000 7f3494ff9640 10 client.4569.objecter successfully canceled tick 2024-03-29T11:21:46.473 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.496+0000 7f3494ff9640 1 -- 172.21.15.170:0/1064590592 >> v1:172.21.15.170:6816/3298994444 conn(0x7f348406de80 legacy=0x7f3484070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:46.473 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.496+0000 7f3494ff9640 10 monclient: shutdown 2024-03-29T11:21:46.473 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.496+0000 7f3494ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:46.473 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.496+0000 7f3494ff9640 1 -- 172.21.15.170:0/1064590592 >> v1:172.21.15.170:6789/0 conn(0x7f349814a700 legacy=0x7f34981d28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:46.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.496+0000 7f3494ff9640 1 -- 172.21.15.170:0/1064590592 shutdown_connections 2024-03-29T11:21:46.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.496+0000 7f3494ff9640 1 -- 172.21.15.170:0/1064590592 >> 172.21.15.170:0/1064590592 conn(0x7f349806b660 msgr2=0x7f34980b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:46.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.496+0000 7f3494ff9640 1 -- 172.21.15.170:0/1064590592 shutdown_connections 2024-03-29T11:21:46.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.496+0000 7f3494ff9640 1 -- 172.21.15.170:0/1064590592 wait complete. 2024-03-29T11:21:46.474 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:46.496+0000 7f3494ff9640 1 librados: shutdown 2024-03-29T11:21:46.490 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:53.297 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:53.430 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.453+0000 7fdf76139640 10 monclient: get_monmap_and_config 2024-03-29T11:21:53.430 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.453+0000 7fdf76139640 10 monclient: build_initial_monmap 2024-03-29T11:21:53.430 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.453+0000 7fdf76139640 10 monclient: monmap: 2024-03-29T11:21:53.430 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:21:53.430 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:53.430 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:21:53.454915+0000 2024-03-29T11:21:53.430 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:21:53.454915+0000 2024-03-29T11:21:53.431 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:53.431 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:53.431 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:21:53.431 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:21:53.431 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:21:53.431 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:53.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.454+0000 7fdf76139640 1 Processor -- start 2024-03-29T11:21:53.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.454+0000 7fdf76139640 1 -- start start 2024-03-29T11:21:53.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.454+0000 7fdf76139640 10 monclient: init 2024-03-29T11:21:53.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.455+0000 7fdf76139640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:53.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.455+0000 7fdf76139640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:53.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.455+0000 7fdf76139640 10 monclient(hunting): picked mon.noname-b con 0x7fdf701457f0 addr v1:172.21.15.170:6790/0 2024-03-29T11:21:53.432 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.455+0000 7fdf76139640 10 monclient(hunting): picked mon.noname-c con 0x7fdf7013f650 addr v1:172.21.15.184:6789/0 2024-03-29T11:21:53.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.455+0000 7fdf76139640 10 monclient(hunting): picked mon.noname-a con 0x7fdf700a9d70 addr v1:172.21.15.170:6789/0 2024-03-29T11:21:53.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.455+0000 7fdf76139640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf700aaef0 con 0x7fdf700a9d70 2024-03-29T11:21:53.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.455+0000 7fdf76139640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf70145bd0 con 0x7fdf7013f650 2024-03-29T11:21:53.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.455+0000 7fdf76139640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf70145eb0 con 0x7fdf701457f0 2024-03-29T11:21:53.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.455+0000 7fdf76139640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:53.433 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.455+0000 7fdf76139640 10 monclient(hunting): authenticate will time out at 1494.767847s 2024-03-29T11:21:53.434 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.456+0000 7fdf6ffff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fdf700a9d70 0x7fdf7014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:48986/0 (socket says 172.21.15.170:48986) 2024-03-29T11:21:53.434 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.456+0000 7fdf6ffff640 1 -- 172.21.15.170:0/1078015241 learned_addr learned my addr 172.21.15.170:0/1078015241 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:21:53.435 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.457+0000 7fdf6e7fc640 1 -- 172.21.15.170:0/1078015241 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500213100 0 0) 0x7fdf700aaef0 con 0x7fdf700a9d70 2024-03-29T11:21:53.435 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.457+0000 7fdf6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:53.435 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.457+0000 7fdf6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:53.435 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.457+0000 7fdf6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:53.435 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.457+0000 7fdf6e7fc640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:21:53.435 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.457+0000 7fdf6e7fc640 1 -- 172.21.15.170:0/1078015241 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf58003520 con 0x7fdf700a9d70 2024-03-29T11:21:53.435 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.457+0000 7fdf6e7fc640 1 -- 172.21.15.170:0/1078015241 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3166046086 0 0) 0x7fdf58003520 con 0x7fdf700a9d70 2024-03-29T11:21:53.435 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.457+0000 7fdf6e7fc640 1 -- 172.21.15.170:0/1078015241 >> v1:172.21.15.170:6790/0 conn(0x7fdf701457f0 legacy=0x7fdf7013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:53.435 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.457+0000 7fdf6e7fc640 1 -- 172.21.15.170:0/1078015241 >> v1:172.21.15.184:6789/0 conn(0x7fdf7013f650 legacy=0x7fdf700b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:53.436 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf6e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:53.436 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf6e7fc640 1 monclient: found mon.noname-a 2024-03-29T11:21:53.436 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:53.436 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf6e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:21:53.436 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf6e7fc640 1 -- 172.21.15.170:0/1078015241 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf70146340 con 0x7fdf700a9d70 2024-03-29T11:21:53.436 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf6e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:53.436 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf6e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:53.436 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:53.436 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf6e7fc640 1 -- 172.21.15.170:0/1078015241 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fdf68001a10 con 0x7fdf700a9d70 2024-03-29T11:21:53.436 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:53.436 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf6e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:53.436 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf6e7fc640 10 monclient: dump: 2024-03-29T11:21:53.436 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:53.437 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:53.437 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:53.437 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:53.437 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:53.437 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:53.437 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:53.437 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:53.437 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:53.437 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:53.437 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf76139640 5 monclient: authenticate success, global_id 4572 2024-03-29T11:21:53.437 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf76139640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:53.439 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf6e7fc640 1 -- 172.21.15.170:0/1078015241 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf68001be0 con 0x7fdf700a9d70 2024-03-29T11:21:53.439 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:53.439 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf6e7fc640 1 -- 172.21.15.170:0/1078015241 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fdf68002030 con 0x7fdf700a9d70 2024-03-29T11:21:53.439 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:53.440 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf6e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:53.440 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf6e7fc640 10 monclient: dump: 2024-03-29T11:21:53.440 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:53.440 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:53.440 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:53.440 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:53.440 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:53.440 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:53.440 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:53.440 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:53.440 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:53.440 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:53.440 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf76139640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:53.440 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.459+0000 7fdf76139640 4 set_mon_vals no callback set 2024-03-29T11:21:53.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.460+0000 7fdf76139640 10 monclient: shutdown 2024-03-29T11:21:53.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.460+0000 7fdf76139640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:53.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.460+0000 7fdf76139640 1 -- 172.21.15.170:0/1078015241 >> v1:172.21.15.170:6789/0 conn(0x7fdf700a9d70 legacy=0x7fdf7014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:53.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.460+0000 7fdf76139640 1 -- 172.21.15.170:0/1078015241 shutdown_connections 2024-03-29T11:21:53.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.460+0000 7fdf76139640 1 -- 172.21.15.170:0/1078015241 >> 172.21.15.170:0/1078015241 conn(0x7fdf70066a00 msgr2=0x7fdf70066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:53.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.460+0000 7fdf76139640 1 -- 172.21.15.170:0/1078015241 shutdown_connections 2024-03-29T11:21:53.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.461+0000 7fdf76139640 1 -- 172.21.15.170:0/1078015241 wait complete. 2024-03-29T11:21:53.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.461+0000 7fdf76139640 10 monclient: build_initial_monmap 2024-03-29T11:21:53.441 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.461+0000 7fdf76139640 10 monclient: monmap: 2024-03-29T11:21:53.441 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:21:53.441 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:53.441 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:21:53.441 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:21:53.441 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:53.442 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:53.442 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:21:53.442 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:21:53.442 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:21:53.442 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:53.442 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.462+0000 7fdf76139640 1 librados: starting msgr at 2024-03-29T11:21:53.442 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.462+0000 7fdf76139640 1 librados: starting objecter 2024-03-29T11:21:53.442 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.462+0000 7fdf76139640 1 Processor -- start 2024-03-29T11:21:53.442 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.462+0000 7fdf76139640 1 -- start start 2024-03-29T11:21:53.442 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.462+0000 7fdf76139640 1 librados: setting wanted keys 2024-03-29T11:21:53.442 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.462+0000 7fdf76139640 1 librados: calling monclient init 2024-03-29T11:21:53.442 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.462+0000 7fdf76139640 10 monclient: init 2024-03-29T11:21:53.448 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.463+0000 7fdf76139640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:53.448 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.463+0000 7fdf76139640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:53.448 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.463+0000 7fdf76139640 10 monclient(hunting): picked mon.noname-c con 0x7fdf700a9d70 addr v1:172.21.15.170:6790/0 2024-03-29T11:21:53.448 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.463+0000 7fdf6f7fe640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fdf700a9d70 0x7fdf701dc360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:44242/0 (socket says 172.21.15.170:44242) 2024-03-29T11:21:53.448 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.464+0000 7fdf76139640 10 monclient(hunting): picked mon.noname-b con 0x7fdf7013f650 addr v1:172.21.15.184:6789/0 2024-03-29T11:21:53.448 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.464+0000 7fdf76139640 10 monclient(hunting): picked mon.noname-a con 0x7fdf701457f0 addr v1:172.21.15.170:6789/0 2024-03-29T11:21:53.448 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.464+0000 7fdf6ffff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fdf701457f0 0x7fdf701e63f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:49314/0 (socket says 172.21.15.170:49314) 2024-03-29T11:21:53.448 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.464+0000 7fdf6ffff640 1 -- 172.21.15.170:0/3616560477 learned_addr learned my addr 172.21.15.170:0/3616560477 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:21:53.448 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.464+0000 7fdf57fff640 10 client.?.objecter ms_handle_connect 0x7fdf700a9d70 2024-03-29T11:21:53.449 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.464+0000 7fdf57fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:53.449 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.464+0000 7fdf57fff640 10 client.?.objecter ms_handle_connect 0x7fdf701457f0 2024-03-29T11:21:53.449 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.464+0000 7fdf57fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:53.449 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.464+0000 7fdf76139640 1 -- 172.21.15.170:0/3616560477 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf70142c30 con 0x7fdf701457f0 2024-03-29T11:21:53.449 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.464+0000 7fdf76139640 1 -- 172.21.15.170:0/3616560477 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf70142dc0 con 0x7fdf7013f650 2024-03-29T11:21:53.449 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.464+0000 7fdf76139640 1 -- 172.21.15.170:0/3616560477 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf701db9b0 con 0x7fdf700a9d70 2024-03-29T11:21:53.449 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.464+0000 7fdf57fff640 1 -- 172.21.15.170:0/3616560477 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329327092 0 0) 0x7fdf70142c30 con 0x7fdf701457f0 2024-03-29T11:21:53.449 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.464+0000 7fdf76139640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:53.449 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.464+0000 7fdf76139640 10 monclient(hunting): authenticate will time out at 1494.776945s 2024-03-29T11:21:53.449 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.464+0000 7fdf57fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:53.449 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:53.449 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:53.449 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:21:53.449 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 1 -- 172.21.15.170:0/3616560477 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf5c004eb0 con 0x7fdf701457f0 2024-03-29T11:21:53.450 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 1 -- 172.21.15.170:0/3616560477 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 328010542 0 0) 0x7fdf5c004eb0 con 0x7fdf701457f0 2024-03-29T11:21:53.450 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 1 -- 172.21.15.170:0/3616560477 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf70142c30 con 0x7fdf701457f0 2024-03-29T11:21:53.450 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 1 -- 172.21.15.170:0/3616560477 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561954773 0 0) 0x7fdf701db9b0 con 0x7fdf700a9d70 2024-03-29T11:21:53.450 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:53.450 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:53.450 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:53.450 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:21:53.450 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 1 -- 172.21.15.170:0/3616560477 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf5c004eb0 con 0x7fdf700a9d70 2024-03-29T11:21:53.450 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 1 -- 172.21.15.170:0/3616560477 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fdf68001ef0 con 0x7fdf701457f0 2024-03-29T11:21:53.450 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:53.450 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:53.450 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 10 monclient(hunting): dump: 2024-03-29T11:21:53.450 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:53.450 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:53.451 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:53.451 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:53.451 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:53.451 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:53.451 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:53.451 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:53.451 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:53.451 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:53.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:53.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 1 -- 172.21.15.170:0/3616560477 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1950994147 0 0) 0x7fdf70142c30 con 0x7fdf701457f0 2024-03-29T11:21:53.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 1 -- 172.21.15.170:0/3616560477 >> v1:172.21.15.170:6790/0 conn(0x7fdf700a9d70 legacy=0x7fdf701dc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:53.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.465+0000 7fdf57fff640 1 -- 172.21.15.170:0/3616560477 >> v1:172.21.15.184:6789/0 conn(0x7fdf7013f650 legacy=0x7fdf7013fda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:53.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.467+0000 7fdf57fff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:53.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.467+0000 7fdf57fff640 1 monclient: found mon.a 2024-03-29T11:21:53.451 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.467+0000 7fdf57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:53.452 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.467+0000 7fdf57fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:21:53.452 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.467+0000 7fdf57fff640 1 -- 172.21.15.170:0/3616560477 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf701dbb40 con 0x7fdf701457f0 2024-03-29T11:21:53.452 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.467+0000 7fdf57fff640 10 monclient: _finish_auth 0 2024-03-29T11:21:53.452 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.467+0000 7fdf57fff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:53.452 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.467+0000 7fdf57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:53.452 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.467+0000 7fdf57fff640 1 -- 172.21.15.170:0/3616560477 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf680020f0 con 0x7fdf701457f0 2024-03-29T11:21:53.452 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.467+0000 7fdf57fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:53.452 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.467+0000 7fdf57fff640 1 -- 172.21.15.170:0/3616560477 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fdf68005110 con 0x7fdf701457f0 2024-03-29T11:21:53.452 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.467+0000 7fdf57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:53.452 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.467+0000 7fdf57fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:53.452 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.467+0000 7fdf57fff640 10 monclient: dump: 2024-03-29T11:21:53.452 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:21:53.452 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:21:53.452 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:53.452 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:21:53.453 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:53.453 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:21:53.453 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:21:53.453 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:21:53.453 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:21:53.453 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:21:53.453 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.467+0000 7fdf6dffb640 4 set_mon_vals no callback set 2024-03-29T11:21:53.453 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.469+0000 7fdf76139640 5 monclient: authenticate success, global_id 4575 2024-03-29T11:21:53.453 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.469+0000 7fdf76139640 10 monclient: _renew_subs 2024-03-29T11:21:53.453 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.469+0000 7fdf76139640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:21:53.453 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.469+0000 7fdf76139640 1 -- 172.21.15.170:0/3616560477 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf70142060 con 0x7fdf701457f0 2024-03-29T11:21:53.453 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.469+0000 7fdf76139640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:53.453 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.469+0000 7fdf76139640 10 monclient: _renew_subs 2024-03-29T11:21:53.453 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.469+0000 7fdf76139640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:21:53.454 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.469+0000 7fdf76139640 1 -- 172.21.15.170:0/3616560477 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf7006e2e0 con 0x7fdf701457f0 2024-03-29T11:21:53.454 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.469+0000 7fdf76139640 1 librados: init done 2024-03-29T11:21:53.455 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.472+0000 7fdf76139640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:53.455 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.472+0000 7fdf76139640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:53.455 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.472+0000 7fdf76139640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:21:53.455 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.472+0000 7fdf76139640 1 -- 172.21.15.170:0/3616560477 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf70075f10 con 0x7fdf701457f0 2024-03-29T11:21:53.455 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.472+0000 7fdf57fff640 1 -- 172.21.15.170:0/3616560477 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fdf6801b0e0 con 0x7fdf701457f0 2024-03-29T11:21:53.455 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.473+0000 7fdf6f7fe640 10 monclient: get_auth_request con 0x7fdf5c06dd90 auth_method 0 2024-03-29T11:21:53.455 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.473+0000 7fdf57fff640 1 -- 172.21.15.170:0/3616560477 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7fdf68084ef0 con 0x7fdf701457f0 2024-03-29T11:21:53.455 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.473+0000 7fdf57fff640 10 client.4575.objecter ms_dispatch 0x7fdf70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:53.456 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.473+0000 7fdf57fff640 3 client.4575.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:53.456 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.473+0000 7fdf57fff640 3 client.4575.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:53.456 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.473+0000 7fdf57fff640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-29T11:21:53.456 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.473+0000 7fdf57fff640 10 client.4575.objecter ms_handle_connect 0x7fdf5c06dd90 2024-03-29T11:21:53.456 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.480+0000 7fdf57fff640 1 -- 172.21.15.170:0/3616560477 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdf680507d0 con 0x7fdf701457f0 2024-03-29T11:21:53.456 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.480+0000 7fdf57fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:53.456 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.480+0000 7fdf57fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:53.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.788+0000 7fdf76139640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:53.765 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.788+0000 7fdf76139640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:53.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.788+0000 7fdf76139640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:21:53.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.788+0000 7fdf76139640 1 -- 172.21.15.170:0/3616560477 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf700b3680 con 0x7fdf701457f0 2024-03-29T11:21:53.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.789+0000 7fdf57fff640 1 -- 172.21.15.170:0/3616560477 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7fdf68052700 con 0x7fdf701457f0 2024-03-29T11:21:53.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.789+0000 7fdf57fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:53.766 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.789+0000 7fdf57fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:53.767 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:21:53.767 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:53.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.792+0000 7fdf76139640 10 librados: watch_flush enter 2024-03-29T11:21:53.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.792+0000 7fdf76139640 10 librados: watch_flush exit 2024-03-29T11:21:53.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.792+0000 7fdf76139640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-29T11:21:53.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.792+0000 7fdf76139640 10 client.4575.objecter successfully canceled tick 2024-03-29T11:21:53.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.792+0000 7fdf76139640 1 -- 172.21.15.170:0/3616560477 >> v1:172.21.15.170:6816/3298994444 conn(0x7fdf5c06dd90 legacy=0x7fdf5c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:53.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.792+0000 7fdf76139640 10 monclient: shutdown 2024-03-29T11:21:53.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.792+0000 7fdf76139640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:53.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.792+0000 7fdf76139640 1 -- 172.21.15.170:0/3616560477 >> v1:172.21.15.170:6789/0 conn(0x7fdf701457f0 legacy=0x7fdf701e63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:53.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.792+0000 7fdf76139640 1 -- 172.21.15.170:0/3616560477 shutdown_connections 2024-03-29T11:21:53.770 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.792+0000 7fdf76139640 1 -- 172.21.15.170:0/3616560477 >> 172.21.15.170:0/3616560477 conn(0x7fdf70066a00 msgr2=0x7fdf700b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:53.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.793+0000 7fdf76139640 1 -- 172.21.15.170:0/3616560477 shutdown_connections 2024-03-29T11:21:53.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.793+0000 7fdf76139640 1 -- 172.21.15.170:0/3616560477 wait complete. 2024-03-29T11:21:53.771 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:21:53.793+0000 7fdf76139640 1 librados: shutdown 2024-03-29T11:21:53.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:00.591 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:00.726 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.750+0000 7ff685f32640 10 monclient: get_monmap_and_config 2024-03-29T11:22:00.726 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.750+0000 7ff685f32640 10 monclient: build_initial_monmap 2024-03-29T11:22:00.727 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.750+0000 7ff685f32640 10 monclient: monmap: 2024-03-29T11:22:00.727 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:22:00.727 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:00.727 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:22:00.751276+0000 2024-03-29T11:22:00.727 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:22:00.751276+0000 2024-03-29T11:22:00.727 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:00.727 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:00.727 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:22:00.727 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:22:00.727 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:22:00.727 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:00.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.751+0000 7ff685f32640 1 Processor -- start 2024-03-29T11:22:00.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.751+0000 7ff685f32640 1 -- start start 2024-03-29T11:22:00.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.751+0000 7ff685f32640 10 monclient: init 2024-03-29T11:22:00.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.752+0000 7ff685f32640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:00.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.752+0000 7ff685f32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:00.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.752+0000 7ff685f32640 10 monclient(hunting): picked mon.noname-a con 0x7ff6801422b0 addr v1:172.21.15.170:6789/0 2024-03-29T11:22:00.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.752+0000 7ff685f32640 10 monclient(hunting): picked mon.noname-c con 0x7ff680142690 addr v1:172.21.15.184:6789/0 2024-03-29T11:22:00.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.752+0000 7ff685f32640 10 monclient(hunting): picked mon.noname-b con 0x7ff6800b17d0 addr v1:172.21.15.170:6790/0 2024-03-29T11:22:00.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.752+0000 7ff67ffff640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7ff6800b17d0 0x7ff6800b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:38850/0 (socket says 172.21.15.170:38850) 2024-03-29T11:22:00.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.753+0000 7ff67ffff640 1 -- 172.21.15.170:0/3872721903 learned_addr learned my addr 172.21.15.170:0/3872721903 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:22:00.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.753+0000 7ff685f32640 1 -- 172.21.15.170:0/3872721903 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff680142c80 con 0x7ff6801422b0 2024-03-29T11:22:00.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.753+0000 7ff685f32640 1 -- 172.21.15.170:0/3872721903 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6800aae40 con 0x7ff680142690 2024-03-29T11:22:00.730 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.753+0000 7ff685f32640 1 -- 172.21.15.170:0/3872721903 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff68013fbd0 con 0x7ff6800b17d0 2024-03-29T11:22:00.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.753+0000 7ff685f32640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:00.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.753+0000 7ff685f32640 10 monclient(hunting): authenticate will time out at 1502.064486s 2024-03-29T11:22:00.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.753+0000 7ff67e7fc640 1 -- 172.21.15.170:0/3872721903 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556152451 0 0) 0x7ff68013fbd0 con 0x7ff6800b17d0 2024-03-29T11:22:00.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.753+0000 7ff67e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:00.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.753+0000 7ff67e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:00.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.753+0000 7ff67e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:00.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.753+0000 7ff67e7fc640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:22:00.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.753+0000 7ff67e7fc640 1 -- 172.21.15.170:0/3872721903 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff668003520 con 0x7ff6800b17d0 2024-03-29T11:22:00.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.755+0000 7ff67e7fc640 1 -- 172.21.15.170:0/3872721903 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3693227116 0 0) 0x7ff668003520 con 0x7ff6800b17d0 2024-03-29T11:22:00.731 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.755+0000 7ff67e7fc640 1 -- 172.21.15.170:0/3872721903 >> v1:172.21.15.184:6789/0 conn(0x7ff680142690 legacy=0x7ff6800b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:00.732 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.755+0000 7ff67e7fc640 1 -- 172.21.15.170:0/3872721903 >> v1:172.21.15.170:6789/0 conn(0x7ff6801422b0 legacy=0x7ff68013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:00.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.756+0000 7ff67e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:00.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.756+0000 7ff67e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:22:00.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.756+0000 7ff67e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:00.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.756+0000 7ff67e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:22:00.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.756+0000 7ff67e7fc640 1 -- 172.21.15.170:0/3872721903 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff680150930 con 0x7ff6800b17d0 2024-03-29T11:22:00.733 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.756+0000 7ff67e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:00.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.756+0000 7ff67e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:00.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.756+0000 7ff67e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:00.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.756+0000 7ff685f32640 5 monclient: authenticate success, global_id 4646 2024-03-29T11:22:00.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.756+0000 7ff685f32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:00.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.756+0000 7ff67e7fc640 1 -- 172.21.15.170:0/3872721903 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7ff678001a10 con 0x7ff6800b17d0 2024-03-29T11:22:00.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.756+0000 7ff67e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:00.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.756+0000 7ff67e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:00.734 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.756+0000 7ff67e7fc640 10 monclient: dump: 2024-03-29T11:22:00.734 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:00.734 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:00.734 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:00.734 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:00.734 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:00.734 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:00.735 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:00.735 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:00.735 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:00.735 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:00.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.757+0000 7ff685f32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:00.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.757+0000 7ff67e7fc640 1 -- 172.21.15.170:0/3872721903 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff678001b70 con 0x7ff6800b17d0 2024-03-29T11:22:00.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.757+0000 7ff67e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:00.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.757+0000 7ff67e7fc640 1 -- 172.21.15.170:0/3872721903 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7ff678001f80 con 0x7ff6800b17d0 2024-03-29T11:22:00.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.757+0000 7ff685f32640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:00.735 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.757+0000 7ff685f32640 4 set_mon_vals no callback set 2024-03-29T11:22:00.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.758+0000 7ff67e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:00.736 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.758+0000 7ff67e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:00.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.758+0000 7ff67e7fc640 10 monclient: dump: 2024-03-29T11:22:00.737 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:00.737 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:00.737 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:00.737 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:00.737 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:00.737 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:00.737 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:00.737 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:00.737 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:00.737 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:00.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.758+0000 7ff685f32640 10 monclient: shutdown 2024-03-29T11:22:00.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.758+0000 7ff685f32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:00.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.758+0000 7ff685f32640 1 -- 172.21.15.170:0/3872721903 >> v1:172.21.15.170:6790/0 conn(0x7ff6800b17d0 legacy=0x7ff6800b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:00.737 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.758+0000 7ff685f32640 1 -- 172.21.15.170:0/3872721903 shutdown_connections 2024-03-29T11:22:00.738 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.758+0000 7ff685f32640 1 -- 172.21.15.170:0/3872721903 >> 172.21.15.170:0/3872721903 conn(0x7ff680066a00 msgr2=0x7ff680066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:00.738 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.758+0000 7ff685f32640 1 -- 172.21.15.170:0/3872721903 shutdown_connections 2024-03-29T11:22:00.738 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.758+0000 7ff685f32640 1 -- 172.21.15.170:0/3872721903 wait complete. 2024-03-29T11:22:00.738 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.759+0000 7ff685f32640 10 monclient: build_initial_monmap 2024-03-29T11:22:00.738 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.759+0000 7ff685f32640 10 monclient: monmap: 2024-03-29T11:22:00.738 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:22:00.738 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:00.738 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:22:00.738 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:22:00.738 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:00.738 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:00.738 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:22:00.738 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:22:00.738 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:22:00.739 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:00.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.760+0000 7ff685f32640 1 librados: starting msgr at 2024-03-29T11:22:00.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.760+0000 7ff685f32640 1 librados: starting objecter 2024-03-29T11:22:00.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.760+0000 7ff685f32640 1 Processor -- start 2024-03-29T11:22:00.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.761+0000 7ff685f32640 1 -- start start 2024-03-29T11:22:00.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.761+0000 7ff685f32640 1 librados: setting wanted keys 2024-03-29T11:22:00.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.761+0000 7ff685f32640 1 librados: calling monclient init 2024-03-29T11:22:00.739 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.761+0000 7ff685f32640 10 monclient: init 2024-03-29T11:22:00.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff685f32640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:00.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff685f32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:00.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff685f32640 10 monclient(hunting): picked mon.noname-a con 0x7ff6800b17d0 addr v1:172.21.15.170:6789/0 2024-03-29T11:22:00.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff685f32640 10 monclient(hunting): picked mon.noname-b con 0x7ff6801422b0 addr v1:172.21.15.184:6789/0 2024-03-29T11:22:00.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff685f32640 10 monclient(hunting): picked mon.noname-c con 0x7ff680142690 addr v1:172.21.15.170:6790/0 2024-03-29T11:22:00.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff67effd640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7ff6801422b0 0x7ff6800ae4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:44956/0 (socket says 172.21.15.170:44956) 2024-03-29T11:22:00.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff67effd640 1 -- 172.21.15.170:0/850059567 learned_addr learned my addr 172.21.15.170:0/850059567 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:22:00.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff685f32640 1 -- 172.21.15.170:0/850059567 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6800aebc0 con 0x7ff6800b17d0 2024-03-29T11:22:00.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff685f32640 1 -- 172.21.15.170:0/850059567 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6800adad0 con 0x7ff6801422b0 2024-03-29T11:22:00.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff667fff640 10 client.?.objecter ms_handle_connect 0x7ff680142690 2024-03-29T11:22:00.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff667fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:00.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff685f32640 1 -- 172.21.15.170:0/850059567 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6800addb0 con 0x7ff680142690 2024-03-29T11:22:00.743 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff685f32640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:00.744 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff685f32640 10 monclient(hunting): authenticate will time out at 1502.073865s 2024-03-29T11:22:00.744 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff667fff640 1 -- 172.21.15.170:0/850059567 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668307409 0 0) 0x7ff6800addb0 con 0x7ff680142690 2024-03-29T11:22:00.744 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff667fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:00.744 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff667fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:00.744 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff667fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:00.744 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff667fff640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:22:00.744 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff667fff640 1 -- 172.21.15.170:0/850059567 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff66c004eb0 con 0x7ff680142690 2024-03-29T11:22:00.744 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff667fff640 10 client.?.objecter ms_handle_connect 0x7ff6801422b0 2024-03-29T11:22:00.744 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.762+0000 7ff667fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:00.744 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.763+0000 7ff667fff640 1 -- 172.21.15.170:0/850059567 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 223426425 0 0) 0x7ff66c004eb0 con 0x7ff680142690 2024-03-29T11:22:00.744 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.763+0000 7ff667fff640 1 -- 172.21.15.170:0/850059567 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6800addb0 con 0x7ff680142690 2024-03-29T11:22:00.744 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.763+0000 7ff667fff640 1 -- 172.21.15.170:0/850059567 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7ff678001f40 con 0x7ff680142690 2024-03-29T11:22:00.744 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.763+0000 7ff667fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:00.744 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.763+0000 7ff667fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:00.745 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.763+0000 7ff667fff640 10 monclient(hunting): dump: 2024-03-29T11:22:00.745 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:00.745 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:00.745 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:00.745 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:00.745 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:00.745 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:00.745 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:00.745 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:00.745 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:00.745 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:00.745 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.763+0000 7ff667fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:00.745 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.763+0000 7ff667fff640 1 -- 172.21.15.170:0/850059567 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2934088173 0 0) 0x7ff6800addb0 con 0x7ff680142690 2024-03-29T11:22:00.745 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.763+0000 7ff667fff640 1 -- 172.21.15.170:0/850059567 >> v1:172.21.15.184:6789/0 conn(0x7ff6801422b0 legacy=0x7ff6800ae4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:00.745 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.763+0000 7ff667fff640 1 -- 172.21.15.170:0/850059567 >> v1:172.21.15.170:6789/0 conn(0x7ff6800b17d0 legacy=0x7ff6800acb00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:00.745 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.764+0000 7ff667fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:00.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.764+0000 7ff667fff640 1 monclient: found mon.c 2024-03-29T11:22:00.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.764+0000 7ff667fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:00.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.764+0000 7ff667fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:00.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.764+0000 7ff667fff640 1 -- 172.21.15.170:0/850059567 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6800ae090 con 0x7ff680142690 2024-03-29T11:22:00.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.764+0000 7ff667fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:00.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.764+0000 7ff667fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:00.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.764+0000 7ff667fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:00.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.764+0000 7ff685f32640 5 monclient: authenticate success, global_id 4649 2024-03-29T11:22:00.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.764+0000 7ff685f32640 10 monclient: _renew_subs 2024-03-29T11:22:00.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.764+0000 7ff685f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:00.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.764+0000 7ff685f32640 1 -- 172.21.15.170:0/850059567 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6800b67f0 con 0x7ff680142690 2024-03-29T11:22:00.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.765+0000 7ff685f32640 10 client.4649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:00.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.765+0000 7ff685f32640 10 monclient: _renew_subs 2024-03-29T11:22:00.746 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.765+0000 7ff685f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:00.747 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.765+0000 7ff685f32640 1 -- 172.21.15.170:0/850059567 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6800b6520 con 0x7ff680142690 2024-03-29T11:22:00.747 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.765+0000 7ff685f32640 1 librados: init done 2024-03-29T11:22:00.747 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.765+0000 7ff667fff640 1 -- 172.21.15.170:0/850059567 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff678004e10 con 0x7ff680142690 2024-03-29T11:22:00.747 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.765+0000 7ff667fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:00.747 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.765+0000 7ff667fff640 1 -- 172.21.15.170:0/850059567 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7ff6780015d0 con 0x7ff680142690 2024-03-29T11:22:00.747 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.765+0000 7ff667fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:00.747 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.765+0000 7ff667fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:00.747 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.765+0000 7ff667fff640 10 monclient: dump: 2024-03-29T11:22:00.747 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:00.747 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:00.747 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:00.747 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:00.747 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:00.747 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:00.747 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:00.747 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:00.748 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:00.748 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:00.748 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.765+0000 7ff67dffb640 4 set_mon_vals no callback set 2024-03-29T11:22:00.748 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.766+0000 7ff685f32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:00.748 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.766+0000 7ff685f32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:00.748 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.766+0000 7ff685f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:00.748 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.766+0000 7ff685f32640 1 -- 172.21.15.170:0/850059567 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff660005080 con 0x7ff680142690 2024-03-29T11:22:00.749 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.767+0000 7ff667fff640 1 -- 172.21.15.170:0/850059567 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7ff67801af90 con 0x7ff680142690 2024-03-29T11:22:00.749 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.769+0000 7ff667fff640 1 -- 172.21.15.170:0/850059567 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1586819124 0 0) 0x7ff678084f50 con 0x7ff680142690 2024-03-29T11:22:00.749 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.769+0000 7ff667fff640 10 client.4649.objecter ms_dispatch 0x7ff680000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:00.749 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.769+0000 7ff667fff640 3 client.4649.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:00.749 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.769+0000 7ff667fff640 3 client.4649.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:00.749 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.769+0000 7ff67f7fe640 10 monclient: get_auth_request con 0x7ff66c06dbd0 auth_method 0 2024-03-29T11:22:00.749 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.769+0000 7ff667fff640 20 client.4649.objecter dump_active .. 0 homeless 2024-03-29T11:22:00.749 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.769+0000 7ff667fff640 10 client.4649.objecter ms_handle_connect 0x7ff66c06dbd0 2024-03-29T11:22:00.751 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.776+0000 7ff667fff640 1 -- 172.21.15.170:0/850059567 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff678050830 con 0x7ff680142690 2024-03-29T11:22:00.752 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.776+0000 7ff667fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:00.752 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:00.776+0000 7ff667fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:01.047 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.071+0000 7ff685f32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:01.048 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.071+0000 7ff685f32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:01.048 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.071+0000 7ff685f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:01.048 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.071+0000 7ff685f32640 1 -- 172.21.15.170:0/850059567 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff660005330 con 0x7ff680142690 2024-03-29T11:22:01.048 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.072+0000 7ff667fff640 1 -- 172.21.15.170:0/850059567 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 388887436) 0x7ff678052760 con 0x7ff680142690 2024-03-29T11:22:01.048 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.072+0000 7ff667fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:01.048 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.072+0000 7ff667fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:01.049 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:22:01.049 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:12:04.169639+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:04.169639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:01.052 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.075+0000 7ff685f32640 10 librados: watch_flush enter 2024-03-29T11:22:01.052 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.075+0000 7ff685f32640 10 librados: watch_flush exit 2024-03-29T11:22:01.052 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.075+0000 7ff685f32640 20 client.4649.objecter shutdown clearing up homeless session... 2024-03-29T11:22:01.052 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.075+0000 7ff685f32640 10 client.4649.objecter successfully canceled tick 2024-03-29T11:22:01.052 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.075+0000 7ff685f32640 1 -- 172.21.15.170:0/850059567 >> v1:172.21.15.170:6816/3298994444 conn(0x7ff66c06dbd0 legacy=0x7ff66c070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:01.052 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.075+0000 7ff685f32640 10 monclient: shutdown 2024-03-29T11:22:01.052 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.075+0000 7ff685f32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:01.052 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.075+0000 7ff685f32640 1 -- 172.21.15.170:0/850059567 >> v1:172.21.15.170:6790/0 conn(0x7ff680142690 legacy=0x7ff6801e1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:01.052 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.076+0000 7ff685f32640 1 -- 172.21.15.170:0/850059567 shutdown_connections 2024-03-29T11:22:01.052 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.076+0000 7ff685f32640 1 -- 172.21.15.170:0/850059567 >> 172.21.15.170:0/850059567 conn(0x7ff680066a00 msgr2=0x7ff68013c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:01.052 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.076+0000 7ff685f32640 1 -- 172.21.15.170:0/850059567 shutdown_connections 2024-03-29T11:22:01.052 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.076+0000 7ff685f32640 1 -- 172.21.15.170:0/850059567 wait complete. 2024-03-29T11:22:01.052 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:01.076+0000 7ff685f32640 1 librados: shutdown 2024-03-29T11:22:01.065 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:07.871 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:08.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.031+0000 7fe9a1f32640 10 monclient: get_monmap_and_config 2024-03-29T11:22:08.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.031+0000 7fe9a1f32640 10 monclient: build_initial_monmap 2024-03-29T11:22:08.008 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.031+0000 7fe9a1f32640 10 monclient: monmap: 2024-03-29T11:22:08.008 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:22:08.008 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:08.008 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:22:08.031749+0000 2024-03-29T11:22:08.009 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:22:08.031749+0000 2024-03-29T11:22:08.009 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:08.009 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:08.009 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:22:08.009 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:22:08.009 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:22:08.009 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:08.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.032+0000 7fe9a1f32640 1 Processor -- start 2024-03-29T11:22:08.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.032+0000 7fe9a1f32640 1 -- start start 2024-03-29T11:22:08.009 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.032+0000 7fe9a1f32640 10 monclient: init 2024-03-29T11:22:08.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.033+0000 7fe9a1f32640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:08.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.033+0000 7fe9a1f32640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:08.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.033+0000 7fe9a1f32640 10 monclient(hunting): picked mon.noname-a con 0x7fe99c0ae250 addr v1:172.21.15.170:6789/0 2024-03-29T11:22:08.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.033+0000 7fe9a1f32640 10 monclient(hunting): picked mon.noname-b con 0x7fe99c13f970 addr v1:172.21.15.170:6790/0 2024-03-29T11:22:08.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.033+0000 7fe9a1f32640 10 monclient(hunting): picked mon.noname-c con 0x7fe99c142800 addr v1:172.21.15.184:6789/0 2024-03-29T11:22:08.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.033+0000 7fe9a1f32640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe99c0afe00 con 0x7fe99c0ae250 2024-03-29T11:22:08.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.033+0000 7fe9a1f32640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe99c142c10 con 0x7fe99c142800 2024-03-29T11:22:08.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.033+0000 7fe9a1f32640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe99c0afb20 con 0x7fe99c13f970 2024-03-29T11:22:08.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.033+0000 7fe9a1f32640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:08.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.033+0000 7fe9a1f32640 10 monclient(hunting): authenticate will time out at 1509.344497s 2024-03-29T11:22:08.011 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.033+0000 7fe99affd640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fe99c13f970 0x7fe99c1420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:53532/0 (socket says 172.21.15.170:53532) 2024-03-29T11:22:08.012 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.033+0000 7fe99affd640 1 -- 172.21.15.170:0/2199976476 learned_addr learned my addr 172.21.15.170:0/2199976476 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:22:08.012 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.033+0000 7fe99a7fc640 1 -- 172.21.15.170:0/2199976476 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852752421 0 0) 0x7fe99c0afb20 con 0x7fe99c13f970 2024-03-29T11:22:08.012 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.033+0000 7fe99a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:08.012 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.033+0000 7fe99a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:08.012 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.033+0000 7fe99a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:08.012 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.033+0000 7fe99a7fc640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:22:08.012 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.033+0000 7fe99a7fc640 1 -- 172.21.15.170:0/2199976476 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe984003520 con 0x7fe99c13f970 2024-03-29T11:22:08.012 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.034+0000 7fe99a7fc640 1 -- 172.21.15.170:0/2199976476 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814785799 0 0) 0x7fe99c142c10 con 0x7fe99c142800 2024-03-29T11:22:08.012 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.034+0000 7fe99a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:08.012 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.034+0000 7fe99a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:08.012 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.034+0000 7fe99a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:08.012 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.034+0000 7fe99a7fc640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:22:08.012 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.034+0000 7fe99a7fc640 1 -- 172.21.15.170:0/2199976476 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe99c0afb20 con 0x7fe99c142800 2024-03-29T11:22:08.012 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.034+0000 7fe99a7fc640 1 -- 172.21.15.170:0/2199976476 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2554238008 0 0) 0x7fe984003520 con 0x7fe99c13f970 2024-03-29T11:22:08.012 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.034+0000 7fe99a7fc640 1 -- 172.21.15.170:0/2199976476 >> v1:172.21.15.184:6789/0 conn(0x7fe99c142800 legacy=0x7fe99c14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:08.012 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.034+0000 7fe99a7fc640 1 -- 172.21.15.170:0/2199976476 >> v1:172.21.15.170:6789/0 conn(0x7fe99c0ae250 legacy=0x7fe99c13f260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:08.013 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.036+0000 7fe99a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:08.013 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.036+0000 7fe99a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:22:08.013 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.036+0000 7fe99a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:08.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.036+0000 7fe99a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:22:08.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.036+0000 7fe99a7fc640 1 -- 172.21.15.170:0/2199976476 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99c0aab80 con 0x7fe99c13f970 2024-03-29T11:22:08.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.036+0000 7fe99a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:08.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.036+0000 7fe99a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:08.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.036+0000 7fe99a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:08.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.036+0000 7fe99a7fc640 1 -- 172.21.15.170:0/2199976476 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe990001690 con 0x7fe99c13f970 2024-03-29T11:22:08.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.036+0000 7fe9a1f32640 5 monclient: authenticate success, global_id 4652 2024-03-29T11:22:08.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.036+0000 7fe99a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:08.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.036+0000 7fe99a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:08.014 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.036+0000 7fe99a7fc640 10 monclient: dump: 2024-03-29T11:22:08.014 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:08.015 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:08.015 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:08.015 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:08.015 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:08.015 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:08.015 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:08.015 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:08.015 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:08.015 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:08.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.036+0000 7fe9a1f32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:08.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.037+0000 7fe99a7fc640 1 -- 172.21.15.170:0/2199976476 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe990001870 con 0x7fe99c13f970 2024-03-29T11:22:08.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.037+0000 7fe99a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:08.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.037+0000 7fe9a1f32640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:08.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.037+0000 7fe99a7fc640 1 -- 172.21.15.170:0/2199976476 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe990001c80 con 0x7fe99c13f970 2024-03-29T11:22:08.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.037+0000 7fe99a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:08.015 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.037+0000 7fe9a1f32640 4 set_mon_vals no callback set 2024-03-29T11:22:08.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.037+0000 7fe99a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:08.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.037+0000 7fe99a7fc640 10 monclient: dump: 2024-03-29T11:22:08.016 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:08.016 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:08.016 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:08.016 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:08.016 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:08.016 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:08.016 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:08.016 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:08.016 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:08.016 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:08.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.038+0000 7fe9a1f32640 10 monclient: shutdown 2024-03-29T11:22:08.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.038+0000 7fe9a1f32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:08.016 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.038+0000 7fe9a1f32640 1 -- 172.21.15.170:0/2199976476 >> v1:172.21.15.170:6790/0 conn(0x7fe99c13f970 legacy=0x7fe99c1420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:08.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.038+0000 7fe9a1f32640 1 -- 172.21.15.170:0/2199976476 shutdown_connections 2024-03-29T11:22:08.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.038+0000 7fe9a1f32640 1 -- 172.21.15.170:0/2199976476 >> 172.21.15.170:0/2199976476 conn(0x7fe99c0b35e0 msgr2=0x7fe99c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:08.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.038+0000 7fe9a1f32640 1 -- 172.21.15.170:0/2199976476 shutdown_connections 2024-03-29T11:22:08.017 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.038+0000 7fe9a1f32640 1 -- 172.21.15.170:0/2199976476 wait complete. 2024-03-29T11:22:08.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.038+0000 7fe9a1f32640 10 monclient: build_initial_monmap 2024-03-29T11:22:08.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.038+0000 7fe9a1f32640 10 monclient: monmap: 2024-03-29T11:22:08.024 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:22:08.024 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:08.024 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:22:08.024 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:22:08.025 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:08.025 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:08.025 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:22:08.025 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:22:08.025 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:22:08.025 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.039+0000 7fe9a1f32640 1 librados: starting msgr at 2024-03-29T11:22:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.039+0000 7fe9a1f32640 1 librados: starting objecter 2024-03-29T11:22:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.039+0000 7fe9a1f32640 1 Processor -- start 2024-03-29T11:22:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.039+0000 7fe9a1f32640 1 -- start start 2024-03-29T11:22:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.039+0000 7fe9a1f32640 1 librados: setting wanted keys 2024-03-29T11:22:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.039+0000 7fe9a1f32640 1 librados: calling monclient init 2024-03-29T11:22:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.039+0000 7fe9a1f32640 10 monclient: init 2024-03-29T11:22:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.040+0000 7fe9a1f32640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:08.025 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.040+0000 7fe9a1f32640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:08.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.040+0000 7fe9a1f32640 10 monclient(hunting): picked mon.noname-c con 0x7fe99c0ae250 addr v1:172.21.15.170:6790/0 2024-03-29T11:22:08.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.040+0000 7fe99b7fe640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fe99c0ae250 0x7fe99c071a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:33392/0 (socket says 172.21.15.170:33392) 2024-03-29T11:22:08.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.040+0000 7fe99b7fe640 1 -- 172.21.15.170:0/3131596255 learned_addr learned my addr 172.21.15.170:0/3131596255 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:22:08.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.040+0000 7fe9a1f32640 10 monclient(hunting): picked mon.noname-a con 0x7fe99c13f970 addr v1:172.21.15.170:6789/0 2024-03-29T11:22:08.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.040+0000 7fe983fff640 10 client.?.objecter ms_handle_connect 0x7fe99c0ae250 2024-03-29T11:22:08.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.040+0000 7fe983fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:08.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.040+0000 7fe9a1f32640 10 monclient(hunting): picked mon.noname-b con 0x7fe99c142800 addr v1:172.21.15.184:6789/0 2024-03-29T11:22:08.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.040+0000 7fe9a1f32640 1 -- 172.21.15.170:0/3131596255 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe99c1388f0 con 0x7fe99c13f970 2024-03-29T11:22:08.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.040+0000 7fe9a1f32640 1 -- 172.21.15.170:0/3131596255 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe99c146070 con 0x7fe99c142800 2024-03-29T11:22:08.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.040+0000 7fe9a1f32640 1 -- 172.21.15.170:0/3131596255 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe99c143c10 con 0x7fe99c0ae250 2024-03-29T11:22:08.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.041+0000 7fe9a1f32640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:08.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.041+0000 7fe9a1f32640 10 monclient(hunting): authenticate will time out at 1509.352066s 2024-03-29T11:22:08.026 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.041+0000 7fe983fff640 1 -- 172.21.15.170:0/3131596255 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159320092 0 0) 0x7fe99c143c10 con 0x7fe99c0ae250 2024-03-29T11:22:08.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.041+0000 7fe983fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:08.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.041+0000 7fe983fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:08.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.041+0000 7fe983fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:08.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.041+0000 7fe983fff640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:22:08.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.041+0000 7fe983fff640 1 -- 172.21.15.170:0/3131596255 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe988004eb0 con 0x7fe99c0ae250 2024-03-29T11:22:08.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.041+0000 7fe983fff640 1 -- 172.21.15.170:0/3131596255 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3611745194 0 0) 0x7fe988004eb0 con 0x7fe99c0ae250 2024-03-29T11:22:08.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.041+0000 7fe983fff640 1 -- 172.21.15.170:0/3131596255 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe99c143c10 con 0x7fe99c0ae250 2024-03-29T11:22:08.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.041+0000 7fe983fff640 1 -- 172.21.15.170:0/3131596255 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe98c001960 con 0x7fe99c0ae250 2024-03-29T11:22:08.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.041+0000 7fe983fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:08.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.041+0000 7fe983fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:08.027 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.041+0000 7fe983fff640 10 monclient(hunting): dump: 2024-03-29T11:22:08.027 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:08.027 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:08.027 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:08.028 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:08.028 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:08.028 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:08.028 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:08.028 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:08.028 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:08.028 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:08.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.041+0000 7fe983fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:08.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.041+0000 7fe983fff640 1 -- 172.21.15.170:0/3131596255 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3592151395 0 0) 0x7fe99c143c10 con 0x7fe99c0ae250 2024-03-29T11:22:08.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.041+0000 7fe983fff640 1 -- 172.21.15.170:0/3131596255 >> v1:172.21.15.184:6789/0 conn(0x7fe99c142800 legacy=0x7fe99c1da640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:08.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.041+0000 7fe983fff640 1 -- 172.21.15.170:0/3131596255 >> v1:172.21.15.170:6789/0 conn(0x7fe99c13f970 legacy=0x7fe99c072110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:08.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.042+0000 7fe983fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:08.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.042+0000 7fe983fff640 1 monclient: found mon.c 2024-03-29T11:22:08.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.042+0000 7fe983fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:08.028 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.042+0000 7fe983fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.042+0000 7fe983fff640 1 -- 172.21.15.170:0/3131596255 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99c143da0 con 0x7fe99c0ae250 2024-03-29T11:22:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.042+0000 7fe983fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.042+0000 7fe983fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.042+0000 7fe983fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.042+0000 7fe9a1f32640 5 monclient: authenticate success, global_id 4655 2024-03-29T11:22:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.042+0000 7fe9a1f32640 10 monclient: _renew_subs 2024-03-29T11:22:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.042+0000 7fe9a1f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.042+0000 7fe9a1f32640 1 -- 172.21.15.170:0/3131596255 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe99c144680 con 0x7fe99c0ae250 2024-03-29T11:22:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.042+0000 7fe9a1f32640 10 client.4655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.042+0000 7fe9a1f32640 10 monclient: _renew_subs 2024-03-29T11:22:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.042+0000 7fe9a1f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.042+0000 7fe9a1f32640 1 -- 172.21.15.170:0/3131596255 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe99c1dad50 con 0x7fe99c0ae250 2024-03-29T11:22:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.042+0000 7fe9a1f32640 1 librados: init done 2024-03-29T11:22:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.044+0000 7fe983fff640 1 -- 172.21.15.170:0/3131596255 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe98c001c80 con 0x7fe99c0ae250 2024-03-29T11:22:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.044+0000 7fe983fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:08.029 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.044+0000 7fe999ffb640 4 set_mon_vals no callback set 2024-03-29T11:22:08.030 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.044+0000 7fe983fff640 1 -- 172.21.15.170:0/3131596255 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fe98c003a40 con 0x7fe99c0ae250 2024-03-29T11:22:08.030 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.045+0000 7fe983fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:08.030 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.045+0000 7fe983fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:08.030 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.045+0000 7fe983fff640 10 monclient: dump: 2024-03-29T11:22:08.030 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:08.030 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:08.030 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:08.030 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:08.030 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:08.030 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:08.030 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:08.030 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:08.030 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:08.030 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:08.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.045+0000 7fe983fff640 1 -- 172.21.15.170:0/3131596255 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fe98c003c60 con 0x7fe99c0ae250 2024-03-29T11:22:08.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.045+0000 7fe9a1f32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:08.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.045+0000 7fe983fff640 1 -- 172.21.15.170:0/3131596255 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4038649127 0 0) 0x7fe98c084b50 con 0x7fe99c0ae250 2024-03-29T11:22:08.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.045+0000 7fe983fff640 10 client.4655.objecter ms_dispatch 0x7fe99c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:22:08.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.045+0000 7fe983fff640 3 client.4655.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:22:08.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.045+0000 7fe983fff640 3 client.4655.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:22:08.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.046+0000 7fe983fff640 20 client.4655.objecter dump_active .. 0 homeless 2024-03-29T11:22:08.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.046+0000 7fe9a1f32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:08.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.046+0000 7fe9a1f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:08.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.046+0000 7fe9a1f32640 1 -- 172.21.15.170:0/3131596255 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe97c005080 con 0x7fe99c0ae250 2024-03-29T11:22:08.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.046+0000 7fe99affd640 10 monclient: get_auth_request con 0x7fe98806db30 auth_method 0 2024-03-29T11:22:08.031 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.046+0000 7fe983fff640 10 client.4655.objecter ms_handle_connect 0x7fe98806db30 2024-03-29T11:22:08.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.053+0000 7fe983fff640 1 -- 172.21.15.170:0/3131596255 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe98c0504b0 con 0x7fe99c0ae250 2024-03-29T11:22:08.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.053+0000 7fe983fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:08.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.053+0000 7fe983fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:08.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.356+0000 7fe9a1f32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:08.333 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.357+0000 7fe9a1f32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:08.333 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.357+0000 7fe9a1f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:08.333 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.357+0000 7fe9a1f32640 1 -- 172.21.15.170:0/3131596255 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe97c005330 con 0x7fe99c0ae250 2024-03-29T11:22:08.333 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.357+0000 7fe983fff640 1 -- 172.21.15.170:0/3131596255 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3062895256) 0x7fe98c0523e0 con 0x7fe99c0ae250 2024-03-29T11:22:08.334 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.357+0000 7fe983fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:08.334 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.357+0000 7fe983fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:08.334 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:22:08.334 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":11,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:22:06.930367+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:22:06.930367+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:08.336 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.360+0000 7fe9a1f32640 10 librados: watch_flush enter 2024-03-29T11:22:08.336 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.360+0000 7fe9a1f32640 10 librados: watch_flush exit 2024-03-29T11:22:08.336 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.360+0000 7fe9a1f32640 20 client.4655.objecter shutdown clearing up homeless session... 2024-03-29T11:22:08.336 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.360+0000 7fe9a1f32640 10 client.4655.objecter successfully canceled tick 2024-03-29T11:22:08.336 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.360+0000 7fe9a1f32640 1 -- 172.21.15.170:0/3131596255 >> v1:172.21.15.170:6816/3298994444 conn(0x7fe98806db30 legacy=0x7fe98806fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:08.336 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.360+0000 7fe9a1f32640 10 monclient: shutdown 2024-03-29T11:22:08.336 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.360+0000 7fe9a1f32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:08.336 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.360+0000 7fe9a1f32640 1 -- 172.21.15.170:0/3131596255 >> v1:172.21.15.170:6790/0 conn(0x7fe99c0ae250 legacy=0x7fe99c071a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:08.336 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.360+0000 7fe9a1f32640 1 -- 172.21.15.170:0/3131596255 shutdown_connections 2024-03-29T11:22:08.336 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.360+0000 7fe9a1f32640 1 -- 172.21.15.170:0/3131596255 >> 172.21.15.170:0/3131596255 conn(0x7fe99c0b35e0 msgr2=0x7fe99c0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:08.337 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.360+0000 7fe9a1f32640 1 -- 172.21.15.170:0/3131596255 shutdown_connections 2024-03-29T11:22:08.337 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.360+0000 7fe9a1f32640 1 -- 172.21.15.170:0/3131596255 wait complete. 2024-03-29T11:22:08.337 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:08.360+0000 7fe9a1f32640 1 librados: shutdown 2024-03-29T11:22:08.350 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:15.157 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:15.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.340+0000 7f22852d9640 10 monclient: get_monmap_and_config 2024-03-29T11:22:15.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.340+0000 7f22852d9640 10 monclient: build_initial_monmap 2024-03-29T11:22:15.316 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.340+0000 7f22852d9640 10 monclient: monmap: 2024-03-29T11:22:15.317 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:22:15.317 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:15.317 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:22:15.340768+0000 2024-03-29T11:22:15.317 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:22:15.340768+0000 2024-03-29T11:22:15.317 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:15.317 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:15.317 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:22:15.317 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:22:15.317 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:22:15.317 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:15.318 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.342+0000 7f22852d9640 1 Processor -- start 2024-03-29T11:22:15.321 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.342+0000 7f22852d9640 1 -- start start 2024-03-29T11:22:15.321 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.342+0000 7f22852d9640 10 monclient: init 2024-03-29T11:22:15.321 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.343+0000 7f22852d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:15.321 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.343+0000 7f22852d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:15.321 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.343+0000 7f22852d9640 10 monclient(hunting): picked mon.noname-c con 0x7f22800a9ac0 addr v1:172.21.15.184:6789/0 2024-03-29T11:22:15.321 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.343+0000 7f22852d9640 10 monclient(hunting): picked mon.noname-b con 0x7f228013ee90 addr v1:172.21.15.170:6790/0 2024-03-29T11:22:15.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.347+0000 7f22852d9640 10 monclient(hunting): picked mon.noname-a con 0x7f22800af900 addr v1:172.21.15.170:6789/0 2024-03-29T11:22:15.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.347+0000 7f22852d9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22800afd10 con 0x7f22800af900 2024-03-29T11:22:15.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.347+0000 7f22852d9640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22800af6f0 con 0x7f22800a9ac0 2024-03-29T11:22:15.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.347+0000 7f22852d9640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22800aac80 con 0x7f228013ee90 2024-03-29T11:22:15.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.347+0000 7f22852d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:15.323 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.347+0000 7f22852d9640 10 monclient(hunting): authenticate will time out at 1516.658322s 2024-03-29T11:22:15.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.349+0000 7f227f7fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f22800af900 0x7f2280149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:57195/0 (socket says 172.21.15.170:57195) 2024-03-29T11:22:15.325 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.349+0000 7f227f7fe640 1 -- 172.21.15.170:0/3952707062 learned_addr learned my addr 172.21.15.170:0/3952707062 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:22:15.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.353+0000 7f227e7fc640 1 -- 172.21.15.170:0/3952707062 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791289928 0 0) 0x7f22800afd10 con 0x7f22800af900 2024-03-29T11:22:15.330 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.353+0000 7f227e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:15.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.353+0000 7f227e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:15.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.353+0000 7f227e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:15.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.353+0000 7f227e7fc640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:22:15.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.353+0000 7f227e7fc640 1 -- 172.21.15.170:0/3952707062 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f226c003520 con 0x7f22800af900 2024-03-29T11:22:15.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.353+0000 7f227e7fc640 1 -- 172.21.15.170:0/3952707062 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557119676 0 0) 0x7f22800af6f0 con 0x7f22800a9ac0 2024-03-29T11:22:15.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.353+0000 7f227e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:15.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.353+0000 7f227e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:15.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.353+0000 7f227e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:15.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.353+0000 7f227e7fc640 10 monclient(hunting): my global_id is 4564 2024-03-29T11:22:15.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.353+0000 7f227e7fc640 1 -- 172.21.15.170:0/3952707062 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22800afd10 con 0x7f22800a9ac0 2024-03-29T11:22:15.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.353+0000 7f227e7fc640 1 -- 172.21.15.170:0/3952707062 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3256197739 0 0) 0x7f22800afd10 con 0x7f22800a9ac0 2024-03-29T11:22:15.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 1 -- 172.21.15.170:0/3952707062 >> v1:172.21.15.170:6790/0 conn(0x7f228013ee90 legacy=0x7f228013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:15.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 1 -- 172.21.15.170:0/3952707062 >> v1:172.21.15.170:6789/0 conn(0x7f22800af900 legacy=0x7f2280149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:15.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:15.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 1 monclient: found mon.noname-c 2024-03-29T11:22:15.331 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:15.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:22:15.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 1 -- 172.21.15.170:0/3952707062 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22800aaf60 con 0x7f22800a9ac0 2024-03-29T11:22:15.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:15.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:15.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:15.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 1 -- 172.21.15.170:0/3952707062 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f2270001670 con 0x7f22800a9ac0 2024-03-29T11:22:15.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:15.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:15.332 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 10 monclient: dump: 2024-03-29T11:22:15.332 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:15.332 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:15.332 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:15.332 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:15.332 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:15.332 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:15.332 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:15.333 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:15.333 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:15.333 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:15.333 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 1 -- 172.21.15.170:0/3952707062 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22700017d0 con 0x7f22800a9ac0 2024-03-29T11:22:15.333 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:15.333 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 1 -- 172.21.15.170:0/3952707062 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f2270001ba0 con 0x7f22800a9ac0 2024-03-29T11:22:15.333 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:15.333 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:15.333 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.354+0000 7f227e7fc640 10 monclient: dump: 2024-03-29T11:22:15.333 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:15.333 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:15.333 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:15.333 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:15.333 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:15.333 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:15.333 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:15.334 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:15.334 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:15.334 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:15.334 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.356+0000 7f22852d9640 5 monclient: authenticate success, global_id 4564 2024-03-29T11:22:15.334 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.356+0000 7f22852d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:15.335 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.356+0000 7f22852d9640 4 set_mon_vals no callback set 2024-03-29T11:22:15.335 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.357+0000 7f22852d9640 10 monclient: shutdown 2024-03-29T11:22:15.335 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.357+0000 7f22852d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:15.335 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.357+0000 7f22852d9640 1 -- 172.21.15.170:0/3952707062 >> v1:172.21.15.184:6789/0 conn(0x7f22800a9ac0 legacy=0x7f22800adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:15.335 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.357+0000 7f22852d9640 1 -- 172.21.15.170:0/3952707062 shutdown_connections 2024-03-29T11:22:15.335 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.357+0000 7f22852d9640 1 -- 172.21.15.170:0/3952707062 >> 172.21.15.170:0/3952707062 conn(0x7f22800b35e0 msgr2=0x7f22800b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:15.335 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.359+0000 7f22852d9640 1 -- 172.21.15.170:0/3952707062 shutdown_connections 2024-03-29T11:22:15.335 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.359+0000 7f22852d9640 1 -- 172.21.15.170:0/3952707062 wait complete. 2024-03-29T11:22:15.337 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.360+0000 7f22852d9640 10 monclient: build_initial_monmap 2024-03-29T11:22:15.337 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.360+0000 7f22852d9640 10 monclient: monmap: 2024-03-29T11:22:15.338 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:22:15.338 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:15.338 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:22:15.338 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:22:15.338 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:15.338 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:15.338 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:22:15.338 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:22:15.338 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:22:15.338 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:15.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.361+0000 7f22852d9640 1 librados: starting msgr at 2024-03-29T11:22:15.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.361+0000 7f22852d9640 1 librados: starting objecter 2024-03-29T11:22:15.338 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.361+0000 7f22852d9640 1 Processor -- start 2024-03-29T11:22:15.339 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.361+0000 7f22852d9640 1 -- start start 2024-03-29T11:22:15.339 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.361+0000 7f22852d9640 1 librados: setting wanted keys 2024-03-29T11:22:15.339 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.361+0000 7f22852d9640 1 librados: calling monclient init 2024-03-29T11:22:15.339 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.361+0000 7f22852d9640 10 monclient: init 2024-03-29T11:22:15.339 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.362+0000 7f22852d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:15.339 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.362+0000 7f22852d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:15.340 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.362+0000 7f22852d9640 10 monclient(hunting): picked mon.noname-b con 0x7f22800af900 addr v1:172.21.15.184:6789/0 2024-03-29T11:22:15.340 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.362+0000 7f22852d9640 10 monclient(hunting): picked mon.noname-a con 0x7f228013ee90 addr v1:172.21.15.170:6789/0 2024-03-29T11:22:15.341 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.365+0000 7f22852d9640 10 monclient(hunting): picked mon.noname-c con 0x7f22801db7c0 addr v1:172.21.15.170:6790/0 2024-03-29T11:22:15.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.365+0000 7f22852d9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22801c8540 con 0x7f228013ee90 2024-03-29T11:22:15.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.365+0000 7f22852d9640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22801dbc50 con 0x7f22800af900 2024-03-29T11:22:15.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.365+0000 7f22852d9640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2280071c40 con 0x7f22801db7c0 2024-03-29T11:22:15.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.365+0000 7f22852d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:15.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.365+0000 7f22852d9640 10 monclient(hunting): authenticate will time out at 1516.676871s 2024-03-29T11:22:15.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.366+0000 7f227f7fe640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f22801db7c0 0x7f2280139330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:51871/0 (socket says 172.21.15.170:51871) 2024-03-29T11:22:15.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.366+0000 7f227f7fe640 1 -- 172.21.15.170:0/3714724357 learned_addr learned my addr 172.21.15.170:0/3714724357 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:22:15.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.366+0000 7f22777fe640 10 client.?.objecter ms_handle_connect 0x7f22801db7c0 2024-03-29T11:22:15.342 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.366+0000 7f22777fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:15.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.366+0000 7f22777fe640 1 -- 172.21.15.170:0/3714724357 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067791726 0 0) 0x7f2280071c40 con 0x7f22801db7c0 2024-03-29T11:22:15.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.366+0000 7f22777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:15.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.366+0000 7f22777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:15.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.366+0000 7f22777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:15.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.366+0000 7f22777fe640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:22:15.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.366+0000 7f22777fe640 1 -- 172.21.15.170:0/3714724357 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2260004eb0 con 0x7f22801db7c0 2024-03-29T11:22:15.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.366+0000 7f22777fe640 10 client.?.objecter ms_handle_connect 0x7f228013ee90 2024-03-29T11:22:15.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.366+0000 7f22777fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:15.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 1 -- 172.21.15.170:0/3714724357 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1974539743 0 0) 0x7f22801c8540 con 0x7f228013ee90 2024-03-29T11:22:15.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:15.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:15.343 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:15.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:22:15.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 1 -- 172.21.15.170:0/3714724357 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2280071c40 con 0x7f228013ee90 2024-03-29T11:22:15.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 1 -- 172.21.15.170:0/3714724357 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3929134038 0 0) 0x7f2260004eb0 con 0x7f22801db7c0 2024-03-29T11:22:15.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 1 -- 172.21.15.170:0/3714724357 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22801c8540 con 0x7f22801db7c0 2024-03-29T11:22:15.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 1 -- 172.21.15.170:0/3714724357 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f2278001c50 con 0x7f22801db7c0 2024-03-29T11:22:15.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:15.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:15.344 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 10 monclient(hunting): dump: 2024-03-29T11:22:15.344 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:15.344 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:15.344 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:15.344 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:15.344 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:15.344 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:15.344 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:15.344 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:15.345 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:15.345 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:15.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:15.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 1 -- 172.21.15.170:0/3714724357 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4033005379 0 0) 0x7f22801c8540 con 0x7f22801db7c0 2024-03-29T11:22:15.345 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 1 -- 172.21.15.170:0/3714724357 >> v1:172.21.15.184:6789/0 conn(0x7f22800af900 legacy=0x7f22801dc170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:15.346 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 1 -- 172.21.15.170:0/3714724357 >> v1:172.21.15.170:6789/0 conn(0x7f228013ee90 legacy=0x7f22801c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:15.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:15.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 1 monclient: found mon.c 2024-03-29T11:22:15.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:15.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:15.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 1 -- 172.21.15.170:0/3714724357 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2280071dd0 con 0x7f22801db7c0 2024-03-29T11:22:15.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:15.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:15.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.367+0000 7f22777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:15.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.368+0000 7f22852d9640 5 monclient: authenticate success, global_id 4661 2024-03-29T11:22:15.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.368+0000 7f22852d9640 10 monclient: _renew_subs 2024-03-29T11:22:15.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.368+0000 7f22852d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:15.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.368+0000 7f22852d9640 1 -- 172.21.15.170:0/3714724357 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2280139a40 con 0x7f22801db7c0 2024-03-29T11:22:15.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.368+0000 7f22852d9640 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:15.347 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.368+0000 7f22852d9640 10 monclient: _renew_subs 2024-03-29T11:22:15.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.368+0000 7f22852d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:15.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.368+0000 7f22852d9640 1 -- 172.21.15.170:0/3714724357 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22801c8ed0 con 0x7f22801db7c0 2024-03-29T11:22:15.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.368+0000 7f22852d9640 1 librados: init done 2024-03-29T11:22:15.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.369+0000 7f22757fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:15.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.369+0000 7f22757fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:15.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.369+0000 7f22757fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:15.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.369+0000 7f22757fa640 1 -- 172.21.15.170:0/3714724357 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22801dbde0 con 0x7f22801db7c0 2024-03-29T11:22:15.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.370+0000 7f22777fe640 1 -- 172.21.15.170:0/3714724357 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2278004b60 con 0x7f22801db7c0 2024-03-29T11:22:15.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.370+0000 7f22777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:15.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.370+0000 7f22777fe640 1 -- 172.21.15.170:0/3714724357 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f2278004fb0 con 0x7f22801db7c0 2024-03-29T11:22:15.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.370+0000 7f22777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:15.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.370+0000 7f22777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:15.348 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.370+0000 7f22777fe640 10 monclient: dump: 2024-03-29T11:22:15.348 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:15.349 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:15.349 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:15.349 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:15.349 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:15.349 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:15.349 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:15.349 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:15.349 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:15.349 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:15.349 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.370+0000 7f227d7fa640 4 set_mon_vals no callback set 2024-03-29T11:22:15.349 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.372+0000 7f22777fe640 1 -- 172.21.15.170:0/3714724357 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f2278005110 con 0x7f22801db7c0 2024-03-29T11:22:15.350 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.375+0000 7f22777fe640 1 -- 172.21.15.170:0/3714724357 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4038649127 0 0) 0x7f2278084cc0 con 0x7f22801db7c0 2024-03-29T11:22:15.352 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.376+0000 7f22777fe640 10 client.4661.objecter ms_dispatch 0x7f2280000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:22:15.352 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.376+0000 7f22777fe640 3 client.4661.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:22:15.352 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.376+0000 7f22777fe640 3 client.4661.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:22:15.352 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.376+0000 7f22777fe640 20 client.4661.objecter dump_active .. 0 homeless 2024-03-29T11:22:15.352 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.376+0000 7f227effd640 10 monclient: get_auth_request con 0x7f226006dd00 auth_method 0 2024-03-29T11:22:15.354 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.378+0000 7f22777fe640 1 -- 172.21.15.170:0/3714724357 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22780505a0 con 0x7f22801db7c0 2024-03-29T11:22:15.354 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.378+0000 7f22777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:15.354 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.378+0000 7f22777fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:15.354 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.379+0000 7f22777fe640 10 client.4661.objecter ms_handle_connect 0x7f226006dd00 2024-03-29T11:22:15.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.705+0000 7f22757fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:15.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.705+0000 7f22757fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:15.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.705+0000 7f22757fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:15.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.705+0000 7f22757fa640 1 -- 172.21.15.170:0/3714724357 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f228013bcd0 con 0x7f22801db7c0 2024-03-29T11:22:15.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.708+0000 7f22777fe640 1 -- 172.21.15.170:0/3714724357 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3062895256) 0x7f22780524d0 con 0x7f22801db7c0 2024-03-29T11:22:15.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.708+0000 7f22777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:15.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.708+0000 7f22777fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:15.687 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:22:15.687 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":11,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:22:06.930367+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:22:06.930367+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:15.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.712+0000 7f22757fa640 10 librados: watch_flush enter 2024-03-29T11:22:15.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.712+0000 7f22757fa640 10 librados: watch_flush exit 2024-03-29T11:22:15.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.712+0000 7f22757fa640 20 client.4661.objecter shutdown clearing up homeless session... 2024-03-29T11:22:15.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.712+0000 7f22757fa640 10 client.4661.objecter successfully canceled tick 2024-03-29T11:22:15.691 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.715+0000 7f22757fa640 1 -- 172.21.15.170:0/3714724357 >> v1:172.21.15.170:6816/3298994444 conn(0x7f226006dd00 legacy=0x7f22600701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:15.691 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.715+0000 7f22757fa640 10 monclient: shutdown 2024-03-29T11:22:15.691 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.715+0000 7f22757fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:15.691 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.715+0000 7f22757fa640 1 -- 172.21.15.170:0/3714724357 >> v1:172.21.15.170:6790/0 conn(0x7f22801db7c0 legacy=0x7f2280139330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:15.691 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.715+0000 7f22757fa640 1 -- 172.21.15.170:0/3714724357 shutdown_connections 2024-03-29T11:22:15.691 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.715+0000 7f22757fa640 1 -- 172.21.15.170:0/3714724357 >> 172.21.15.170:0/3714724357 conn(0x7f22800b35e0 msgr2=0x7f22800b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:15.692 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.716+0000 7f22757fa640 1 -- 172.21.15.170:0/3714724357 shutdown_connections 2024-03-29T11:22:15.692 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.716+0000 7f22757fa640 1 -- 172.21.15.170:0/3714724357 wait complete. 2024-03-29T11:22:15.692 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:15.716+0000 7f22757fa640 1 librados: shutdown 2024-03-29T11:22:15.709 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:22.516 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:22.665 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.688+0000 7f45b2eb3640 10 monclient: get_monmap_and_config 2024-03-29T11:22:22.665 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.688+0000 7f45b2eb3640 10 monclient: build_initial_monmap 2024-03-29T11:22:22.665 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.688+0000 7f45b2eb3640 10 monclient: monmap: 2024-03-29T11:22:22.665 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:22:22.665 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:22.666 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:22:22.690165+0000 2024-03-29T11:22:22.666 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:22:22.690165+0000 2024-03-29T11:22:22.666 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:22.666 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:22.666 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:22:22.666 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:22:22.666 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:22:22.666 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:22.667 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.689+0000 7f45b2eb3640 1 Processor -- start 2024-03-29T11:22:22.667 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.689+0000 7f45b2eb3640 1 -- start start 2024-03-29T11:22:22.667 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.690+0000 7f45b2eb3640 10 monclient: init 2024-03-29T11:22:22.667 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.690+0000 7f45b2eb3640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:22.667 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.690+0000 7f45b2eb3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:22.667 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.690+0000 7f45b2eb3640 10 monclient(hunting): picked mon.noname-a con 0x7f45a40389e0 addr v1:172.21.15.170:6789/0 2024-03-29T11:22:22.667 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.690+0000 7f45b2eb3640 10 monclient(hunting): picked mon.noname-c con 0x7f45a4038dc0 addr v1:172.21.15.184:6789/0 2024-03-29T11:22:22.667 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.690+0000 7f45b2eb3640 10 monclient(hunting): picked mon.noname-b con 0x7f45a4030660 addr v1:172.21.15.170:6790/0 2024-03-29T11:22:22.667 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.690+0000 7f45b2eb3640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a40c54c0 con 0x7f45a40389e0 2024-03-29T11:22:22.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.690+0000 7f45b2eb3640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a4036430 con 0x7f45a4038dc0 2024-03-29T11:22:22.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.690+0000 7f45b2eb3640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a4039840 con 0x7f45a4030660 2024-03-29T11:22:22.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.690+0000 7f45b2eb3640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:22.668 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.690+0000 7f45b2eb3640 10 monclient(hunting): authenticate will time out at 1524.002909s 2024-03-29T11:22:22.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.696+0000 7f45b26b2640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f45a4030660 0x7f45a40d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:46775/0 (socket says 172.21.15.170:46775) 2024-03-29T11:22:22.673 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.696+0000 7f45b26b2640 1 -- 172.21.15.170:0/345028831 learned_addr learned my addr 172.21.15.170:0/345028831 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:22:22.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 1 -- 172.21.15.170:0/345028831 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393605837 0 0) 0x7f45a4039840 con 0x7f45a4030660 2024-03-29T11:22:22.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 10 monclient(hunting): _negotiate 2024-03-29T11:22:22.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:22.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:22.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:22:22.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 1 -- 172.21.15.170:0/345028831 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4598003520 con 0x7f45a4030660 2024-03-29T11:22:22.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 1 -- 172.21.15.170:0/345028831 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 101438326 0 0) 0x7f4598003520 con 0x7f45a4030660 2024-03-29T11:22:22.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 1 -- 172.21.15.170:0/345028831 >> v1:172.21.15.184:6789/0 conn(0x7f45a4038dc0 legacy=0x7f45a40cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:22.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 1 -- 172.21.15.170:0/345028831 >> v1:172.21.15.170:6789/0 conn(0x7f45a40389e0 legacy=0x7f45a4033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:22.676 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 10 monclient: _finish_hunting 0 2024-03-29T11:22:22.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 1 monclient: found mon.noname-b 2024-03-29T11:22:22.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:22.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:22:22.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 1 -- 172.21.15.170:0/345028831 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a4035690 con 0x7f45a4030660 2024-03-29T11:22:22.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 10 monclient: _finish_auth 0 2024-03-29T11:22:22.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 10 monclient: _check_auth_tickets 2024-03-29T11:22:22.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:22.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 1 -- 172.21.15.170:0/345028831 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f45ac04f730 con 0x7f45a4030660 2024-03-29T11:22:22.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:22.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:22.677 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b0eaf640 10 monclient: dump: 2024-03-29T11:22:22.677 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:22.677 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:22.677 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:22.678 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:22.678 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:22.678 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:22.678 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:22.678 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:22.678 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:22.678 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:22.678 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b2eb3640 5 monclient: authenticate success, global_id 4664 2024-03-29T11:22:22.678 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.699+0000 7f45b2eb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:22.678 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.700+0000 7f45b0eaf640 1 -- 172.21.15.170:0/345028831 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45ac0008d0 con 0x7f45a4030660 2024-03-29T11:22:22.678 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.700+0000 7f45b0eaf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:22.679 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.700+0000 7f45b0eaf640 1 -- 172.21.15.170:0/345028831 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f45ac0758b0 con 0x7f45a4030660 2024-03-29T11:22:22.679 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.700+0000 7f45b0eaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:22.679 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.700+0000 7f45b0eaf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:22.679 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.700+0000 7f45b0eaf640 10 monclient: dump: 2024-03-29T11:22:22.679 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:22.679 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:22.679 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:22.679 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:22.679 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:22.679 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:22.679 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:22.679 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:22.679 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:22.679 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:22.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.701+0000 7f45b2eb3640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:22.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.701+0000 7f45b2eb3640 4 set_mon_vals no callback set 2024-03-29T11:22:22.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.702+0000 7f45b2eb3640 10 monclient: shutdown 2024-03-29T11:22:22.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.702+0000 7f45b2eb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:22.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.702+0000 7f45b2eb3640 1 -- 172.21.15.170:0/345028831 >> v1:172.21.15.170:6790/0 conn(0x7f45a4030660 legacy=0x7f45a40d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:22.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.703+0000 7f45b2eb3640 1 -- 172.21.15.170:0/345028831 shutdown_connections 2024-03-29T11:22:22.680 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.703+0000 7f45b2eb3640 1 -- 172.21.15.170:0/345028831 >> 172.21.15.170:0/345028831 conn(0x7f45a4005c00 msgr2=0x7f45a4006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:22.681 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.704+0000 7f45b2eb3640 1 -- 172.21.15.170:0/345028831 shutdown_connections 2024-03-29T11:22:22.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.705+0000 7f45b2eb3640 1 -- 172.21.15.170:0/345028831 wait complete. 2024-03-29T11:22:22.682 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.706+0000 7f45b2eb3640 10 monclient: build_initial_monmap 2024-03-29T11:22:22.683 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.706+0000 7f45b2eb3640 10 monclient: monmap: 2024-03-29T11:22:22.683 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:22:22.683 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:22.683 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:22:22.683 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:22:22.683 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:22.683 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:22.683 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:22:22.683 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:22:22.683 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:22:22.683 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:22.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.706+0000 7f45b2eb3640 1 librados: starting msgr at 2024-03-29T11:22:22.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.706+0000 7f45b2eb3640 1 librados: starting objecter 2024-03-29T11:22:22.684 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.706+0000 7f45b2eb3640 1 Processor -- start 2024-03-29T11:22:22.685 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.706+0000 7f45b2eb3640 1 -- start start 2024-03-29T11:22:22.685 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.706+0000 7f45b2eb3640 1 librados: setting wanted keys 2024-03-29T11:22:22.685 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.706+0000 7f45b2eb3640 1 librados: calling monclient init 2024-03-29T11:22:22.685 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.706+0000 7f45b2eb3640 10 monclient: init 2024-03-29T11:22:22.685 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.707+0000 7f45b2eb3640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:22.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.707+0000 7f45b2eb3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:22.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.707+0000 7f45b2eb3640 10 monclient(hunting): picked mon.noname-c con 0x7f45a4030660 addr v1:172.21.15.170:6790/0 2024-03-29T11:22:22.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.707+0000 7f45b2eb3640 10 monclient(hunting): picked mon.noname-a con 0x7f45a40389e0 addr v1:172.21.15.170:6789/0 2024-03-29T11:22:22.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.707+0000 7f45b2eb3640 10 monclient(hunting): picked mon.noname-b con 0x7f45a402f8f0 addr v1:172.21.15.184:6789/0 2024-03-29T11:22:22.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.707+0000 7f45b2eb3640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a402f180 con 0x7f45a40389e0 2024-03-29T11:22:22.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.707+0000 7f45b2eb3640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a402f460 con 0x7f45a402f8f0 2024-03-29T11:22:22.688 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.707+0000 7f45b2eb3640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a402d860 con 0x7f45a4030660 2024-03-29T11:22:22.691 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.707+0000 7f45b2eb3640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:22.691 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.707+0000 7f45b2eb3640 10 monclient(hunting): authenticate will time out at 1524.019817s 2024-03-29T11:22:22.691 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.712+0000 7f45b1eb1640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f45a4030660 0x7f45a402e620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:46877/0 (socket says 172.21.15.170:46877) 2024-03-29T11:22:22.691 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.712+0000 7f45b1eb1640 1 -- 172.21.15.170:0/3083477626 learned_addr learned my addr 172.21.15.170:0/3083477626 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:22:22.691 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.712+0000 7f45a27fc640 10 client.?.objecter ms_handle_connect 0x7f45a4030660 2024-03-29T11:22:22.691 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.712+0000 7f45a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:22.691 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.712+0000 7f45a27fc640 1 -- 172.21.15.170:0/3083477626 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672203106 0 0) 0x7f45a402d860 con 0x7f45a4030660 2024-03-29T11:22:22.691 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.712+0000 7f45a27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:22.691 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.712+0000 7f45a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:22.691 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.712+0000 7f45a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:22.691 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.712+0000 7f45a27fc640 10 monclient(hunting): my global_id is 4667 2024-03-29T11:22:22.691 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.712+0000 7f45a27fc640 1 -- 172.21.15.170:0/3083477626 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4590004eb0 con 0x7f45a4030660 2024-03-29T11:22:22.692 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.712+0000 7f45a27fc640 1 -- 172.21.15.170:0/3083477626 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 881802578 0 0) 0x7f4590004eb0 con 0x7f45a4030660 2024-03-29T11:22:22.692 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.712+0000 7f45a27fc640 1 -- 172.21.15.170:0/3083477626 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45a402d860 con 0x7f45a4030660 2024-03-29T11:22:22.692 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.712+0000 7f45a27fc640 1 -- 172.21.15.170:0/3083477626 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f45a8004080 con 0x7f45a4030660 2024-03-29T11:22:22.692 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.712+0000 7f45a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:22.692 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.712+0000 7f45a27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:22.692 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.712+0000 7f45a27fc640 10 monclient(hunting): dump: 2024-03-29T11:22:22.692 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:22.692 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:22.692 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:22.692 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:22.692 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:22.692 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:22.692 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:22.692 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:22.692 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:22.692 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:22.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.712+0000 7f45a27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:22.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.713+0000 7f45a27fc640 1 -- 172.21.15.170:0/3083477626 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 896329478 0 0) 0x7f45a402d860 con 0x7f45a4030660 2024-03-29T11:22:22.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.713+0000 7f45a27fc640 1 -- 172.21.15.170:0/3083477626 >> v1:172.21.15.184:6789/0 conn(0x7f45a402f8f0 legacy=0x7f45a416b660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:22.693 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.713+0000 7f45a27fc640 1 -- 172.21.15.170:0/3083477626 >> v1:172.21.15.170:6789/0 conn(0x7f45a40389e0 legacy=0x7f45a402d150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:22.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.715+0000 7f45a27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:22.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.715+0000 7f45a27fc640 1 monclient: found mon.c 2024-03-29T11:22:22.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.715+0000 7f45a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:22.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.715+0000 7f45a27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:22.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.715+0000 7f45a27fc640 1 -- 172.21.15.170:0/3083477626 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a4030060 con 0x7f45a4030660 2024-03-29T11:22:22.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.715+0000 7f45a27fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:22.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.715+0000 7f45a27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:22.696 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.715+0000 7f45a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:22.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.715+0000 7f45b2eb3640 5 monclient: authenticate success, global_id 4667 2024-03-29T11:22:22.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.715+0000 7f45b2eb3640 10 monclient: _renew_subs 2024-03-29T11:22:22.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.715+0000 7f45b2eb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:22.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.715+0000 7f45b2eb3640 1 -- 172.21.15.170:0/3083477626 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a40c7590 con 0x7f45a4030660 2024-03-29T11:22:22.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.715+0000 7f45b2eb3640 10 client.4667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:22.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.715+0000 7f45b2eb3640 10 monclient: _renew_subs 2024-03-29T11:22:22.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.715+0000 7f45b2eb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:22.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.715+0000 7f45b2eb3640 1 -- 172.21.15.170:0/3083477626 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a416cd80 con 0x7f45a4030660 2024-03-29T11:22:22.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.715+0000 7f45b2eb3640 1 librados: init done 2024-03-29T11:22:22.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.716+0000 7f45a27fc640 1 -- 172.21.15.170:0/3083477626 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45a80042e0 con 0x7f45a4030660 2024-03-29T11:22:22.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.716+0000 7f45a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:22.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.716+0000 7f45a3fff640 4 set_mon_vals no callback set 2024-03-29T11:22:22.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.717+0000 7f45a27fc640 1 -- 172.21.15.170:0/3083477626 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f45a8001320 con 0x7f45a4030660 2024-03-29T11:22:22.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.717+0000 7f45a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:22.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.717+0000 7f45a27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:22.697 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.717+0000 7f45a27fc640 10 monclient: dump: 2024-03-29T11:22:22.698 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:22.698 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:22.698 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:22.698 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:22.698 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:22.698 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:22.698 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:22.698 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:22.698 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:22.698 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:22.698 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.718+0000 7f45b2eb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:22.698 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.718+0000 7f45b2eb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:22.698 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.718+0000 7f45b2eb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:22.698 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.718+0000 7f45b2eb3640 1 -- 172.21.15.170:0/3083477626 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4594005080 con 0x7f45a4030660 2024-03-29T11:22:22.699 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.720+0000 7f45a27fc640 1 -- 172.21.15.170:0/3083477626 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f45a8001480 con 0x7f45a4030660 2024-03-29T11:22:22.700 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.721+0000 7f45b16b0640 10 monclient: get_auth_request con 0x7f459006dc10 auth_method 0 2024-03-29T11:22:22.700 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.722+0000 7f45a27fc640 10 client.4667.objecter ms_handle_connect 0x7f459006dc10 2024-03-29T11:22:22.700 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.722+0000 7f45a27fc640 1 -- 172.21.15.170:0/3083477626 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4038649127 0 0) 0x7f45a8084bf0 con 0x7f45a4030660 2024-03-29T11:22:22.700 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.722+0000 7f45a27fc640 10 client.4667.objecter ms_dispatch 0x7f45ac000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:22:22.700 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.722+0000 7f45a27fc640 3 client.4667.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:22:22.700 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.722+0000 7f45a27fc640 3 client.4667.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:22:22.700 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.722+0000 7f45a27fc640 20 client.4667.objecter dump_active .. 0 homeless 2024-03-29T11:22:22.705 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.728+0000 7f45a27fc640 1 -- 172.21.15.170:0/3083477626 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f45a80504d0 con 0x7f45a4030660 2024-03-29T11:22:22.705 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.728+0000 7f45a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:22.705 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:22.729+0000 7f45a27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:23.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.055+0000 7f45b2eb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:23.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.055+0000 7f45b2eb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:23.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.055+0000 7f45b2eb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:23.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.055+0000 7f45b2eb3640 1 -- 172.21.15.170:0/3083477626 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4594005330 con 0x7f45a4030660 2024-03-29T11:22:23.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.058+0000 7f45a27fc640 1 -- 172.21.15.170:0/3083477626 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3062895256) 0x7f45a8052400 con 0x7f45a4030660 2024-03-29T11:22:23.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.058+0000 7f45a27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:23.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.058+0000 7f45a27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:23.036 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:22:23.036 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":11,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:22:06.930367+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:22:06.930367+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:23.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.062+0000 7f45b2eb3640 10 librados: watch_flush enter 2024-03-29T11:22:23.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.062+0000 7f45b2eb3640 10 librados: watch_flush exit 2024-03-29T11:22:23.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.062+0000 7f45b2eb3640 20 client.4667.objecter shutdown clearing up homeless session... 2024-03-29T11:22:23.039 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.062+0000 7f45b2eb3640 10 client.4667.objecter successfully canceled tick 2024-03-29T11:22:23.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.062+0000 7f45b2eb3640 1 -- 172.21.15.170:0/3083477626 >> v1:172.21.15.170:6816/3298994444 conn(0x7f459006dc10 legacy=0x7f45900700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:23.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.062+0000 7f45b2eb3640 10 monclient: shutdown 2024-03-29T11:22:23.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.062+0000 7f45b2eb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:23.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.062+0000 7f45b2eb3640 1 -- 172.21.15.170:0/3083477626 >> v1:172.21.15.170:6790/0 conn(0x7f45a4030660 legacy=0x7f45a402e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:23.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.062+0000 7f45b2eb3640 1 -- 172.21.15.170:0/3083477626 shutdown_connections 2024-03-29T11:22:23.040 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.063+0000 7f45b2eb3640 1 -- 172.21.15.170:0/3083477626 >> 172.21.15.170:0/3083477626 conn(0x7f45a4005c00 msgr2=0x7f45a4034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:23.041 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.065+0000 7f45b2eb3640 1 -- 172.21.15.170:0/3083477626 shutdown_connections 2024-03-29T11:22:23.041 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.065+0000 7f45b2eb3640 1 -- 172.21.15.170:0/3083477626 wait complete. 2024-03-29T11:22:23.042 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:23.065+0000 7f45b2eb3640 1 librados: shutdown 2024-03-29T11:22:23.059 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:29.866 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:30.018 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.041+0000 7f33b8ac1640 10 monclient: get_monmap_and_config 2024-03-29T11:22:30.018 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.041+0000 7f33b8ac1640 10 monclient: build_initial_monmap 2024-03-29T11:22:30.018 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.041+0000 7f33b8ac1640 10 monclient: monmap: 2024-03-29T11:22:30.018 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:22:30.018 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:30.018 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:22:30.042898+0000 2024-03-29T11:22:30.018 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:22:30.042898+0000 2024-03-29T11:22:30.019 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:30.019 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:30.019 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:22:30.019 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:22:30.019 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:22:30.019 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:30.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.042+0000 7f33b8ac1640 1 Processor -- start 2024-03-29T11:22:30.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.042+0000 7f33b8ac1640 1 -- start start 2024-03-29T11:22:30.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.042+0000 7f33b8ac1640 10 monclient: init 2024-03-29T11:22:30.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.043+0000 7f33b8ac1640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:30.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.043+0000 7f33b8ac1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:30.020 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.043+0000 7f33b8ac1640 10 monclient(hunting): picked mon.noname-b con 0x7f33ac006e30 addr v1:172.21.15.170:6790/0 2024-03-29T11:22:30.023 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.046+0000 7f33b8ac1640 10 monclient(hunting): picked mon.noname-c con 0x7f33ac035e00 addr v1:172.21.15.184:6789/0 2024-03-29T11:22:30.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.046+0000 7f33b8ac1640 10 monclient(hunting): picked mon.noname-a con 0x7f33ac0c9670 addr v1:172.21.15.170:6789/0 2024-03-29T11:22:30.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.047+0000 7f33b8ac1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33ac035970 con 0x7f33ac0c9670 2024-03-29T11:22:30.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.047+0000 7f33b8ac1640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33ac036280 con 0x7f33ac035e00 2024-03-29T11:22:30.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.047+0000 7f33b8ac1640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33ac035640 con 0x7f33ac006e30 2024-03-29T11:22:30.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.047+0000 7f33b8ac1640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:30.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.047+0000 7f33b8ac1640 10 monclient(hunting): authenticate will time out at 1531.359060s 2024-03-29T11:22:30.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.047+0000 7f33b3fff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f33ac0c9670 0x7f33ac0d7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:39596/0 (socket says 172.21.15.170:39596) 2024-03-29T11:22:30.024 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.047+0000 7f33b3fff640 1 -- 172.21.15.170:0/877354303 learned_addr learned my addr 172.21.15.170:0/877354303 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:22:30.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 1 -- 172.21.15.170:0/877354303 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453100476 0 0) 0x7f33ac036280 con 0x7f33ac035e00 2024-03-29T11:22:30.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:30.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:30.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:30.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient(hunting): my global_id is 4567 2024-03-29T11:22:30.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 1 -- 172.21.15.170:0/877354303 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33a4003520 con 0x7f33ac035e00 2024-03-29T11:22:30.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 1 -- 172.21.15.170:0/877354303 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638303363 0 0) 0x7f33ac035970 con 0x7f33ac0c9670 2024-03-29T11:22:30.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:30.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:30.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:30.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:22:30.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 1 -- 172.21.15.170:0/877354303 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33ac036280 con 0x7f33ac0c9670 2024-03-29T11:22:30.032 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 1 -- 172.21.15.170:0/877354303 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566158402 0 0) 0x7f33ac035640 con 0x7f33ac006e30 2024-03-29T11:22:30.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:30.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:30.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:30.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient(hunting): my global_id is 4670 2024-03-29T11:22:30.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 1 -- 172.21.15.170:0/877354303 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33ac035970 con 0x7f33ac006e30 2024-03-29T11:22:30.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 1 -- 172.21.15.170:0/877354303 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1031306739 0 0) 0x7f33a4003520 con 0x7f33ac035e00 2024-03-29T11:22:30.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 1 -- 172.21.15.170:0/877354303 >> v1:172.21.15.170:6790/0 conn(0x7f33ac006e30 legacy=0x7f33ac0cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:30.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 1 -- 172.21.15.170:0/877354303 >> v1:172.21.15.170:6789/0 conn(0x7f33ac0c9670 legacy=0x7f33ac0d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:30.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:30.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 1 monclient: found mon.noname-c 2024-03-29T11:22:30.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:30.033 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:22:30.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 1 -- 172.21.15.170:0/877354303 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ac034e20 con 0x7f33ac035e00 2024-03-29T11:22:30.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:30.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:30.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:30.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 1 -- 172.21.15.170:0/877354303 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f33a8001690 con 0x7f33ac035e00 2024-03-29T11:22:30.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:30.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:30.034 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient: dump: 2024-03-29T11:22:30.034 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:30.034 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:30.034 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:30.034 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:30.034 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:30.034 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:30.035 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:30.035 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:30.035 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:30.035 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:30.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b8ac1640 5 monclient: authenticate success, global_id 4567 2024-03-29T11:22:30.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b8ac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:30.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 1 -- 172.21.15.170:0/877354303 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33a80017f0 con 0x7f33ac035e00 2024-03-29T11:22:30.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:30.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 1 -- 172.21.15.170:0/877354303 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f33a8001c00 con 0x7f33ac035e00 2024-03-29T11:22:30.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b8ac1640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:30.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:30.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:30.035 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b27fc640 10 monclient: dump: 2024-03-29T11:22:30.035 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:30.035 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:30.035 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:30.036 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:30.036 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:30.036 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:30.036 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:30.036 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:30.036 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:30.036 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:30.036 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.049+0000 7f33b8ac1640 4 set_mon_vals no callback set 2024-03-29T11:22:30.036 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.050+0000 7f33b8ac1640 10 monclient: shutdown 2024-03-29T11:22:30.036 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.050+0000 7f33b8ac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:30.036 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.050+0000 7f33b8ac1640 1 -- 172.21.15.170:0/877354303 >> v1:172.21.15.184:6789/0 conn(0x7f33ac035e00 legacy=0x7f33ac0c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:30.036 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.052+0000 7f33b8ac1640 1 -- 172.21.15.170:0/877354303 shutdown_connections 2024-03-29T11:22:30.036 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.052+0000 7f33b8ac1640 1 -- 172.21.15.170:0/877354303 >> 172.21.15.170:0/877354303 conn(0x7f33ac005c00 msgr2=0x7f33ac006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:30.036 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.053+0000 7f33b8ac1640 1 -- 172.21.15.170:0/877354303 shutdown_connections 2024-03-29T11:22:30.036 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.053+0000 7f33b8ac1640 1 -- 172.21.15.170:0/877354303 wait complete. 2024-03-29T11:22:30.037 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.054+0000 7f33b8ac1640 10 monclient: build_initial_monmap 2024-03-29T11:22:30.037 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.054+0000 7f33b8ac1640 10 monclient: monmap: 2024-03-29T11:22:30.037 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:22:30.037 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:30.037 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:22:30.037 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:22:30.037 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:30.037 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:30.037 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:22:30.037 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:22:30.037 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:22:30.037 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:30.037 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.054+0000 7f33b8ac1640 1 librados: starting msgr at 2024-03-29T11:22:30.037 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.054+0000 7f33b8ac1640 1 librados: starting objecter 2024-03-29T11:22:30.037 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.054+0000 7f33b8ac1640 1 Processor -- start 2024-03-29T11:22:30.038 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.055+0000 7f33b8ac1640 1 -- start start 2024-03-29T11:22:30.038 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.055+0000 7f33b8ac1640 1 librados: setting wanted keys 2024-03-29T11:22:30.038 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.055+0000 7f33b8ac1640 1 librados: calling monclient init 2024-03-29T11:22:30.038 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.055+0000 7f33b8ac1640 10 monclient: init 2024-03-29T11:22:30.043 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.056+0000 7f33b8ac1640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:30.044 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.056+0000 7f33b8ac1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:30.044 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.056+0000 7f33b8ac1640 10 monclient(hunting): picked mon.noname-a con 0x7f33ac006e30 addr v1:172.21.15.170:6789/0 2024-03-29T11:22:30.044 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.056+0000 7f33b37fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f33ac006e30 0x7f33ac0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:40772/0 (socket says 172.21.15.170:40772) 2024-03-29T11:22:30.044 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.056+0000 7f33b37fe640 1 -- 172.21.15.170:0/97412933 learned_addr learned my addr 172.21.15.170:0/97412933 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:22:30.044 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.057+0000 7f33b8ac1640 10 monclient(hunting): picked mon.noname-c con 0x7f33ac035e00 addr v1:172.21.15.170:6790/0 2024-03-29T11:22:30.044 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.057+0000 7f33b8ac1640 10 monclient(hunting): picked mon.noname-b con 0x7f33ac0c9670 addr v1:172.21.15.184:6789/0 2024-03-29T11:22:30.044 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.057+0000 7f33b8ac1640 1 -- 172.21.15.170:0/97412933 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33ac031260 con 0x7f33ac006e30 2024-03-29T11:22:30.044 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.057+0000 7f33b8ac1640 1 -- 172.21.15.170:0/97412933 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33ac0c2810 con 0x7f33ac0c9670 2024-03-29T11:22:30.044 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.057+0000 7f33b8ac1640 1 -- 172.21.15.170:0/97412933 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33ac0d0060 con 0x7f33ac035e00 2024-03-29T11:22:30.044 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.057+0000 7f33b8ac1640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:30.044 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.057+0000 7f33b8ac1640 10 monclient(hunting): authenticate will time out at 1531.369736s 2024-03-29T11:22:30.044 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.058+0000 7f339ffff640 10 client.?.objecter ms_handle_connect 0x7f33ac006e30 2024-03-29T11:22:30.044 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.058+0000 7f339ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:30.044 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.058+0000 7f339ffff640 1 -- 172.21.15.170:0/97412933 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957490331 0 0) 0x7f33ac031260 con 0x7f33ac006e30 2024-03-29T11:22:30.045 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.058+0000 7f339ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:30.045 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.058+0000 7f339ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:30.045 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.058+0000 7f339ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:30.045 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.058+0000 7f339ffff640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:22:30.045 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.058+0000 7f339ffff640 1 -- 172.21.15.170:0/97412933 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3398004eb0 con 0x7f33ac006e30 2024-03-29T11:22:30.045 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.059+0000 7f339ffff640 1 -- 172.21.15.170:0/97412933 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3760621151 0 0) 0x7f3398004eb0 con 0x7f33ac006e30 2024-03-29T11:22:30.045 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.059+0000 7f339ffff640 1 -- 172.21.15.170:0/97412933 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33ac031260 con 0x7f33ac006e30 2024-03-29T11:22:30.045 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.059+0000 7f339ffff640 1 -- 172.21.15.170:0/97412933 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f33a0001a10 con 0x7f33ac006e30 2024-03-29T11:22:30.045 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.059+0000 7f339ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:30.045 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.059+0000 7f339ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:30.045 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.059+0000 7f339ffff640 10 monclient(hunting): dump: 2024-03-29T11:22:30.045 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:30.045 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:30.045 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:30.046 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:30.046 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:30.046 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:30.046 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:30.046 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:30.046 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:30.046 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:30.046 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.060+0000 7f339ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:30.046 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.060+0000 7f339ffff640 1 -- 172.21.15.170:0/97412933 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2432807116 0 0) 0x7f33ac031260 con 0x7f33ac006e30 2024-03-29T11:22:30.046 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.060+0000 7f339ffff640 1 -- 172.21.15.170:0/97412933 >> v1:172.21.15.170:6790/0 conn(0x7f33ac035e00 legacy=0x7f33ac0cd280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:30.046 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.061+0000 7f339ffff640 1 -- 172.21.15.170:0/97412933 >> v1:172.21.15.184:6789/0 conn(0x7f33ac0c9670 legacy=0x7f33ac16b920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:30.046 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.061+0000 7f339ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:30.046 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.061+0000 7f339ffff640 1 monclient: found mon.a 2024-03-29T11:22:30.046 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.061+0000 7f339ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:30.047 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.061+0000 7f339ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:22:30.047 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.061+0000 7f339ffff640 1 -- 172.21.15.170:0/97412933 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ac0cef50 con 0x7f33ac006e30 2024-03-29T11:22:30.047 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.061+0000 7f339ffff640 10 monclient: _finish_auth 0 2024-03-29T11:22:30.047 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.061+0000 7f339ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:30.047 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.061+0000 7f339ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:30.047 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.061+0000 7f33b8ac1640 5 monclient: authenticate success, global_id 4590 2024-03-29T11:22:30.047 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.061+0000 7f33b8ac1640 10 monclient: _renew_subs 2024-03-29T11:22:30.047 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.061+0000 7f33b8ac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:22:30.047 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.061+0000 7f33b8ac1640 1 -- 172.21.15.170:0/97412933 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33ac0cf1c0 con 0x7f33ac006e30 2024-03-29T11:22:30.047 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.061+0000 7f33b8ac1640 10 client.4590.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:30.047 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.062+0000 7f33b8ac1640 10 monclient: _renew_subs 2024-03-29T11:22:30.047 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.062+0000 7f33b8ac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:22:30.047 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.062+0000 7f33b8ac1640 1 -- 172.21.15.170:0/97412933 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33ac0cd990 con 0x7f33ac006e30 2024-03-29T11:22:30.047 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.062+0000 7f33b8ac1640 1 librados: init done 2024-03-29T11:22:30.048 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.062+0000 7f339ffff640 1 -- 172.21.15.170:0/97412933 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33a0001b70 con 0x7f33ac006e30 2024-03-29T11:22:30.048 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.062+0000 7f339ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:30.048 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.062+0000 7f339ffff640 1 -- 172.21.15.170:0/97412933 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f33a0003a60 con 0x7f33ac006e30 2024-03-29T11:22:30.048 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.062+0000 7f339ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:30.048 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.062+0000 7f339ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:30.048 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.062+0000 7f339ffff640 10 monclient: dump: 2024-03-29T11:22:30.048 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:30.048 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:30.048 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:30.048 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:30.048 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:30.048 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:30.048 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:30.048 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:30.049 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:30.049 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:30.049 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.063+0000 7f33b1ffb640 4 set_mon_vals no callback set 2024-03-29T11:22:30.049 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.063+0000 7f33b8ac1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:30.049 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.064+0000 7f339ffff640 1 -- 172.21.15.170:0/97412933 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f33a0003bc0 con 0x7f33ac006e30 2024-03-29T11:22:30.049 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.064+0000 7f33b8ac1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:30.049 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.067+0000 7f33b8ac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:22:30.049 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.067+0000 7f33b8ac1640 1 -- 172.21.15.170:0/97412933 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3394005080 con 0x7f33ac006e30 2024-03-29T11:22:30.050 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.067+0000 7f33b2ffd640 10 monclient: get_auth_request con 0x7f339806daa0 auth_method 0 2024-03-29T11:22:30.050 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.067+0000 7f339ffff640 1 -- 172.21.15.170:0/97412933 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4038649127 0 0) 0x7f33a0084c10 con 0x7f33ac006e30 2024-03-29T11:22:30.050 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.067+0000 7f339ffff640 10 client.4590.objecter ms_dispatch 0x7f33b4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:22:30.050 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.067+0000 7f339ffff640 3 client.4590.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:22:30.050 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.067+0000 7f339ffff640 3 client.4590.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:22:30.050 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.067+0000 7f339ffff640 20 client.4590.objecter dump_active .. 0 homeless 2024-03-29T11:22:30.050 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.068+0000 7f339ffff640 10 client.4590.objecter ms_handle_connect 0x7f339806daa0 2024-03-29T11:22:30.052 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.076+0000 7f339ffff640 1 -- 172.21.15.170:0/97412933 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33a00504f0 con 0x7f33ac006e30 2024-03-29T11:22:30.052 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.076+0000 7f339ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:30.053 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.076+0000 7f339ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:30.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.407+0000 7f33b8ac1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:30.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.407+0000 7f33b8ac1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:30.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.407+0000 7f33b8ac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:22:30.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.407+0000 7f33b8ac1640 1 -- 172.21.15.170:0/97412933 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3394005330 con 0x7f33ac006e30 2024-03-29T11:22:30.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.409+0000 7f339ffff640 1 -- 172.21.15.170:0/97412933 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3062895256) 0x7f33a0052420 con 0x7f33ac006e30 2024-03-29T11:22:30.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.409+0000 7f339ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:30.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.409+0000 7f339ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:30.387 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:22:30.387 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":11,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:22:06.930367+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:22:06.930367+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:30.392 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.414+0000 7f33b8ac1640 10 librados: watch_flush enter 2024-03-29T11:22:30.392 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.415+0000 7f33b8ac1640 10 librados: watch_flush exit 2024-03-29T11:22:30.392 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.415+0000 7f33b8ac1640 20 client.4590.objecter shutdown clearing up homeless session... 2024-03-29T11:22:30.392 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.415+0000 7f33b8ac1640 10 client.4590.objecter successfully canceled tick 2024-03-29T11:22:30.392 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.415+0000 7f33b8ac1640 1 -- 172.21.15.170:0/97412933 >> v1:172.21.15.170:6816/3298994444 conn(0x7f339806daa0 legacy=0x7f339806ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:30.392 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.415+0000 7f33b8ac1640 10 monclient: shutdown 2024-03-29T11:22:30.392 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.415+0000 7f33b8ac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:30.392 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.415+0000 7f33b8ac1640 1 -- 172.21.15.170:0/97412933 >> v1:172.21.15.170:6789/0 conn(0x7f33ac006e30 legacy=0x7f33ac0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:30.392 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.415+0000 7f33b8ac1640 1 -- 172.21.15.170:0/97412933 shutdown_connections 2024-03-29T11:22:30.392 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.415+0000 7f33b8ac1640 1 -- 172.21.15.170:0/97412933 >> 172.21.15.170:0/97412933 conn(0x7f33ac005c00 msgr2=0x7f33ac0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:30.392 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.415+0000 7f33b8ac1640 1 -- 172.21.15.170:0/97412933 shutdown_connections 2024-03-29T11:22:30.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.415+0000 7f33b8ac1640 1 -- 172.21.15.170:0/97412933 wait complete. 2024-03-29T11:22:30.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:30.416+0000 7f33b8ac1640 1 librados: shutdown 2024-03-29T11:22:30.410 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:37.216 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.403+0000 7efd10814640 10 monclient: get_monmap_and_config 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.403+0000 7efd10814640 10 monclient: build_initial_monmap 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.403+0000 7efd10814640 10 monclient: monmap: 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:22:37.404705+0000 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:22:37.404705+0000 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.404+0000 7efd10814640 1 Processor -- start 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.404+0000 7efd10814640 1 -- start start 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.404+0000 7efd10814640 10 monclient: init 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.405+0000 7efd10814640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.405+0000 7efd10814640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.405+0000 7efd10814640 10 monclient(hunting): picked mon.noname-b con 0x7efd0c0b0770 addr v1:172.21.15.170:6790/0 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.405+0000 7efd10814640 10 monclient(hunting): picked mon.noname-a con 0x7efd0c14a700 addr v1:172.21.15.170:6789/0 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.405+0000 7efd10814640 10 monclient(hunting): picked mon.noname-c con 0x7efd0c0ac5a0 addr v1:172.21.15.184:6789/0 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.405+0000 7efd10814640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0c0aba70 con 0x7efd0c14a700 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.405+0000 7efd10814640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0c0aecb0 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.405+0000 7efd10814640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0c0af150 con 0x7efd0c0b0770 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.405+0000 7efd10814640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.405+0000 7efd10814640 10 monclient(hunting): authenticate will time out at 1538.717680s 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.405+0000 7efd0ad9d640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7efd0c0ac5a0 0x7efd0c151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:50710/0 (socket says 172.21.15.170:50710) 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.405+0000 7efd0ad9d640 1 -- 172.21.15.170:0/1786656127 learned_addr learned my addr 172.21.15.170:0/1786656127 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:22:37.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.406+0000 7efd0959a640 1 -- 172.21.15.170:0/1786656127 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856798216 0 0) 0x7efd0c0aba70 con 0x7efd0c14a700 2024-03-29T11:22:37.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.406+0000 7efd0959a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:37.384 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.406+0000 7efd0959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.406+0000 7efd0959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.406+0000 7efd0959a640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.406+0000 7efd0959a640 1 -- 172.21.15.170:0/1786656127 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcf4003520 con 0x7efd0c14a700 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.406+0000 7efd0959a640 1 -- 172.21.15.170:0/1786656127 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994061504 0 0) 0x7efd0c0aecb0 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.406+0000 7efd0959a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.406+0000 7efd0959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.406+0000 7efd0959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.406+0000 7efd0959a640 10 monclient(hunting): my global_id is 4570 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.406+0000 7efd0959a640 1 -- 172.21.15.170:0/1786656127 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd0c0aba70 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.406+0000 7efd0959a640 1 -- 172.21.15.170:0/1786656127 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675277346 0 0) 0x7efd0c0af150 con 0x7efd0c0b0770 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.406+0000 7efd0959a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.406+0000 7efd0959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.406+0000 7efd0959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.406+0000 7efd0959a640 10 monclient(hunting): my global_id is 4673 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.406+0000 7efd0959a640 1 -- 172.21.15.170:0/1786656127 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcf4004a30 con 0x7efd0c0b0770 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd0959a640 1 -- 172.21.15.170:0/1786656127 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4126936996 0 0) 0x7efcf4003520 con 0x7efd0c14a700 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd0959a640 1 -- 172.21.15.170:0/1786656127 >> v1:172.21.15.170:6790/0 conn(0x7efd0c0b0770 legacy=0x7efd0c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd0959a640 1 -- 172.21.15.170:0/1786656127 >> v1:172.21.15.184:6789/0 conn(0x7efd0c0ac5a0 legacy=0x7efd0c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd0959a640 10 monclient: _finish_hunting 0 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd0959a640 1 monclient: found mon.noname-a 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd0959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd0959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.170:6789/0 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd0959a640 1 -- 172.21.15.170:0/1786656127 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0c0ac9d0 con 0x7efd0c14a700 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd0959a640 10 monclient: _finish_auth 0 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd0959a640 10 monclient: _check_auth_tickets 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd0959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd10814640 5 monclient: authenticate success, global_id 4593 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd10814640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd0959a640 1 -- 172.21.15.170:0/1786656127 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7efcfc001690 con 0x7efd0c14a700 2024-03-29T11:22:37.387 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:37.387 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd0959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:37.387 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd0959a640 10 monclient: dump: 2024-03-29T11:22:37.387 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:37.387 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:37.387 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:37.387 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:37.387 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:37.387 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:37.387 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:37.387 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:37.387 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:37.387 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:37.387 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd10814640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:37.387 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd0959a640 1 -- 172.21.15.170:0/1786656127 <== mon.0 v1:172.21.15.170:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efcfc001870 con 0x7efd0c14a700 2024-03-29T11:22:37.387 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd0959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:37.388 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd10814640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:37.388 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.407+0000 7efd10814640 4 set_mon_vals no callback set 2024-03-29T11:22:37.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.408+0000 7efd0959a640 1 -- 172.21.15.170:0/1786656127 <== mon.0 v1:172.21.15.170:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7efcfc001690 con 0x7efd0c14a700 2024-03-29T11:22:37.393 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.408+0000 7efd0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:37.394 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.408+0000 7efd0959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:37.394 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.408+0000 7efd0959a640 10 monclient: dump: 2024-03-29T11:22:37.394 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:37.394 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:37.394 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:37.394 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:37.394 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:37.394 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:37.394 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:37.394 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:37.394 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:37.394 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:37.394 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.408+0000 7efd10814640 10 monclient: shutdown 2024-03-29T11:22:37.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.408+0000 7efd10814640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:37.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.408+0000 7efd10814640 1 -- 172.21.15.170:0/1786656127 >> v1:172.21.15.170:6789/0 conn(0x7efd0c14a700 legacy=0x7efd0c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:37.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.409+0000 7efd10814640 1 -- 172.21.15.170:0/1786656127 shutdown_connections 2024-03-29T11:22:37.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.409+0000 7efd10814640 1 -- 172.21.15.170:0/1786656127 >> 172.21.15.170:0/1786656127 conn(0x7efd0c06b660 msgr2=0x7efd0c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:37.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.409+0000 7efd10814640 1 -- 172.21.15.170:0/1786656127 shutdown_connections 2024-03-29T11:22:37.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.409+0000 7efd10814640 1 -- 172.21.15.170:0/1786656127 wait complete. 2024-03-29T11:22:37.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.411+0000 7efd10814640 10 monclient: build_initial_monmap 2024-03-29T11:22:37.395 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.411+0000 7efd10814640 10 monclient: monmap: 2024-03-29T11:22:37.395 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:22:37.395 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:37.395 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:22:37.395 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:22:37.395 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:37.395 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:37.396 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:22:37.396 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:22:37.396 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:22:37.396 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:37.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.411+0000 7efd10814640 1 librados: starting msgr at 2024-03-29T11:22:37.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.411+0000 7efd10814640 1 librados: starting objecter 2024-03-29T11:22:37.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.411+0000 7efd10814640 1 Processor -- start 2024-03-29T11:22:37.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.411+0000 7efd10814640 1 -- start start 2024-03-29T11:22:37.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.411+0000 7efd10814640 1 librados: setting wanted keys 2024-03-29T11:22:37.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.411+0000 7efd10814640 1 librados: calling monclient init 2024-03-29T11:22:37.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.411+0000 7efd10814640 10 monclient: init 2024-03-29T11:22:37.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.412+0000 7efd10814640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:37.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.412+0000 7efd10814640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:37.396 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.412+0000 7efd10814640 10 monclient(hunting): picked mon.noname-b con 0x7efd0c0ac5a0 addr v1:172.21.15.184:6789/0 2024-03-29T11:22:37.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.412+0000 7efd10814640 10 monclient(hunting): picked mon.noname-c con 0x7efd0c0b0770 addr v1:172.21.15.170:6790/0 2024-03-29T11:22:37.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efd10814640 10 monclient(hunting): picked mon.noname-a con 0x7efd0c14a700 addr v1:172.21.15.170:6789/0 2024-03-29T11:22:37.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efd10814640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0c1dc020 con 0x7efd0c14a700 2024-03-29T11:22:37.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efd0a59c640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7efd0c0ac5a0 0x7efd0c1239d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.170:51204/0 (socket says 172.21.15.170:51204) 2024-03-29T11:22:37.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efd0a59c640 1 -- 172.21.15.170:0/2982884968 learned_addr learned my addr 172.21.15.170:0/2982884968 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:22:37.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efd10814640 1 -- 172.21.15.170:0/2982884968 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0c143d80 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efcfaffd640 10 client.?.objecter ms_handle_connect 0x7efd0c0ac5a0 2024-03-29T11:22:37.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efcfaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:37.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efd10814640 1 -- 172.21.15.170:0/2982884968 --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0c144b40 con 0x7efd0c0b0770 2024-03-29T11:22:37.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efcfaffd640 10 client.?.objecter ms_handle_connect 0x7efd0c0b0770 2024-03-29T11:22:37.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efcfaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:37.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efcfaffd640 10 client.?.objecter ms_handle_connect 0x7efd0c14a700 2024-03-29T11:22:37.397 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efcfaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:37.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efd10814640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:37.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efd10814640 10 monclient(hunting): authenticate will time out at 1538.725473s 2024-03-29T11:22:37.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efcfaffd640 1 -- 172.21.15.170:0/2982884968 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570007362 0 0) 0x7efd0c143d80 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efcfaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:37.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efcfaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:37.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efcfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:37.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efcfaffd640 10 monclient(hunting): my global_id is 4573 2024-03-29T11:22:37.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.413+0000 7efcfaffd640 1 -- 172.21.15.170:0/2982884968 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcec004eb0 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.414+0000 7efcfaffd640 1 -- 172.21.15.170:0/2982884968 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 499107237 0 0) 0x7efcec004eb0 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.414+0000 7efcfaffd640 1 -- 172.21.15.170:0/2982884968 --> v1:172.21.15.184:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd0c143d80 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.414+0000 7efcfaffd640 1 -- 172.21.15.170:0/2982884968 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677507341 0 0) 0x7efd0c1dc020 con 0x7efd0c14a700 2024-03-29T11:22:37.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.414+0000 7efcfaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:37.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.414+0000 7efcfaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:37.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.414+0000 7efcfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:37.398 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.414+0000 7efcfaffd640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:22:37.399 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.414+0000 7efcfaffd640 1 -- 172.21.15.170:0/2982884968 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcec004eb0 con 0x7efd0c14a700 2024-03-29T11:22:37.399 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.414+0000 7efcfaffd640 1 -- 172.21.15.170:0/2982884968 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7efd00001a20 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.399 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.414+0000 7efcfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:37.399 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.414+0000 7efcfaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:37.399 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.414+0000 7efcfaffd640 10 monclient(hunting): dump: 2024-03-29T11:22:37.399 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:37.399 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:37.399 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:37.399 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:37.399 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:37.399 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:37.399 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:37.399 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:37.399 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:37.399 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:37.399 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.414+0000 7efcfaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:37.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.414+0000 7efcfaffd640 1 -- 172.21.15.170:0/2982884968 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542653952 0 0) 0x7efd0c144b40 con 0x7efd0c0b0770 2024-03-29T11:22:37.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.414+0000 7efcfaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:37.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.414+0000 7efcfaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:37.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.414+0000 7efcfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:37.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.414+0000 7efcfaffd640 10 monclient(hunting): my global_id is 4676 2024-03-29T11:22:37.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.414+0000 7efcfaffd640 1 -- 172.21.15.170:0/2982884968 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd0c1dc020 con 0x7efd0c0b0770 2024-03-29T11:22:37.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.415+0000 7efcfaffd640 1 -- 172.21.15.170:0/2982884968 <== mon.1 v1:172.21.15.184:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2778686602 0 0) 0x7efd0c143d80 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.415+0000 7efcfaffd640 1 -- 172.21.15.170:0/2982884968 >> v1:172.21.15.170:6790/0 conn(0x7efd0c0b0770 legacy=0x7efd0c142f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:37.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.415+0000 7efcfaffd640 1 -- 172.21.15.170:0/2982884968 >> v1:172.21.15.170:6789/0 conn(0x7efd0c14a700 legacy=0x7efd0c143670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:37.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.415+0000 7efcfaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:37.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.415+0000 7efcfaffd640 1 monclient: found mon.b 2024-03-29T11:22:37.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.415+0000 7efcfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:37.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.415+0000 7efcfaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:22:37.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.415+0000 7efcfaffd640 1 -- 172.21.15.170:0/2982884968 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0c1dba70 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.400 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.415+0000 7efcfaffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:37.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.415+0000 7efcfaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:37.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.415+0000 7efcfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:37.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.416+0000 7efd10814640 5 monclient: authenticate success, global_id 4573 2024-03-29T11:22:37.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.416+0000 7efd10814640 10 monclient: _renew_subs 2024-03-29T11:22:37.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.416+0000 7efd10814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:22:37.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.416+0000 7efd10814640 1 -- 172.21.15.170:0/2982884968 --> v1:172.21.15.184:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd0c1dbc60 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.416+0000 7efd10814640 10 client.4573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:37.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.416+0000 7efd10814640 10 monclient: _renew_subs 2024-03-29T11:22:37.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.416+0000 7efd10814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:22:37.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.416+0000 7efd10814640 1 -- 172.21.15.170:0/2982884968 --> v1:172.21.15.184:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd0c144210 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.416+0000 7efd10814640 1 librados: init done 2024-03-29T11:22:37.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.416+0000 7efcfaffd640 1 -- 172.21.15.170:0/2982884968 <== mon.1 v1:172.21.15.184:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd00001d60 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.416+0000 7efcfaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:37.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.416+0000 7efcfaffd640 1 -- 172.21.15.170:0/2982884968 <== mon.1 v1:172.21.15.184:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7efd00003a40 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.416+0000 7efcfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:37.401 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.416+0000 7efcfaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:37.402 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.416+0000 7efcfaffd640 10 monclient: dump: 2024-03-29T11:22:37.402 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:37.402 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:37.402 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:37.402 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:37.402 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:37.402 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:37.402 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:37.402 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:37.402 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:37.402 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:37.402 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.416+0000 7efd08d99640 4 set_mon_vals no callback set 2024-03-29T11:22:37.403 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.419+0000 7efd10814640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:37.403 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.420+0000 7efd10814640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:37.403 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.420+0000 7efd10814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:22:37.403 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.420+0000 7efd10814640 1 -- 172.21.15.170:0/2982884968 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf0005080 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.403 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.421+0000 7efcfaffd640 1 -- 172.21.15.170:0/2982884968 <== mon.1 v1:172.21.15.184:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7efd00003ba0 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.403 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.423+0000 7efcfaffd640 1 -- 172.21.15.170:0/2982884968 <== mon.1 v1:172.21.15.184:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4038649127 0 0) 0x7efd00084b40 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.403 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.423+0000 7efcfaffd640 10 client.4573.objecter ms_dispatch 0x7efd0c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:22:37.403 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.423+0000 7efcfaffd640 3 client.4573.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:22:37.404 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.423+0000 7efcfaffd640 3 client.4573.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:22:37.404 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.423+0000 7efcfaffd640 20 client.4573.objecter dump_active .. 0 homeless 2024-03-29T11:22:37.404 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.423+0000 7efd09d9b640 10 monclient: get_auth_request con 0x7efcec06db00 auth_method 0 2024-03-29T11:22:37.404 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.423+0000 7efcfaffd640 10 client.4573.objecter ms_handle_connect 0x7efcec06db00 2024-03-29T11:22:37.406 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.429+0000 7efcfaffd640 1 -- 172.21.15.170:0/2982884968 <== mon.1 v1:172.21.15.184:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd00050420 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.406 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.429+0000 7efcfaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:37.406 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.429+0000 7efcfaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:37.798 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.821+0000 7efd10814640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:37.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.821+0000 7efd10814640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:37.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.821+0000 7efd10814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.184:6789/0 2024-03-29T11:22:37.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.821+0000 7efd10814640 1 -- 172.21.15.170:0/2982884968 --> v1:172.21.15.184:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efcf0005330 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.823+0000 7efcfaffd640 1 -- 172.21.15.170:0/2982884968 <== mon.1 v1:172.21.15.184:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3062895256) 0x7efd00052350 con 0x7efd0c0ac5a0 2024-03-29T11:22:37.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.823+0000 7efcfaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:37.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.823+0000 7efcfaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:37.800 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:22:37.800 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":11,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:22:06.930367+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:22:06.930367+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:37.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.826+0000 7efd10814640 10 librados: watch_flush enter 2024-03-29T11:22:37.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.826+0000 7efd10814640 10 librados: watch_flush exit 2024-03-29T11:22:37.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.826+0000 7efd10814640 20 client.4573.objecter shutdown clearing up homeless session... 2024-03-29T11:22:37.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.826+0000 7efd10814640 10 client.4573.objecter successfully canceled tick 2024-03-29T11:22:37.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.826+0000 7efd10814640 1 -- 172.21.15.170:0/2982884968 >> v1:172.21.15.170:6816/3298994444 conn(0x7efcec06db00 legacy=0x7efcec06ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:37.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.826+0000 7efd10814640 10 monclient: shutdown 2024-03-29T11:22:37.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.826+0000 7efd10814640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:37.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.826+0000 7efd10814640 1 -- 172.21.15.170:0/2982884968 >> v1:172.21.15.184:6789/0 conn(0x7efd0c0ac5a0 legacy=0x7efd0c1239d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:37.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.826+0000 7efd10814640 1 -- 172.21.15.170:0/2982884968 shutdown_connections 2024-03-29T11:22:37.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.826+0000 7efd10814640 1 -- 172.21.15.170:0/2982884968 >> 172.21.15.170:0/2982884968 conn(0x7efd0c06b660 msgr2=0x7efd0c0b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:37.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.826+0000 7efd10814640 1 -- 172.21.15.170:0/2982884968 shutdown_connections 2024-03-29T11:22:37.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.826+0000 7efd10814640 1 -- 172.21.15.170:0/2982884968 wait complete. 2024-03-29T11:22:37.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:37.826+0000 7efd10814640 1 librados: shutdown 2024-03-29T11:22:37.819 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:44.626 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:44.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.807+0000 7fd167fff640 10 monclient: get_monmap_and_config 2024-03-29T11:22:44.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.807+0000 7fd167fff640 10 monclient: build_initial_monmap 2024-03-29T11:22:44.785 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.807+0000 7fd167fff640 10 monclient: monmap: 2024-03-29T11:22:44.785 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:22:44.785 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:44.785 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:22:44.807992+0000 2024-03-29T11:22:44.785 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:22:44.807992+0000 2024-03-29T11:22:44.785 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:44.786 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:44.786 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:22:44.786 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:22:44.786 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:22:44.786 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:44.786 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.808+0000 7fd167fff640 1 Processor -- start 2024-03-29T11:22:44.786 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.809+0000 7fd167fff640 1 -- start start 2024-03-29T11:22:44.786 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.809+0000 7fd167fff640 10 monclient: init 2024-03-29T11:22:44.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.809+0000 7fd167fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:44.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.809+0000 7fd167fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:44.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.809+0000 7fd167fff640 10 monclient(hunting): picked mon.noname-a con 0x7fd160006410 addr v1:172.21.15.170:6789/0 2024-03-29T11:22:44.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.809+0000 7fd167fff640 10 monclient(hunting): picked mon.noname-b con 0x7fd16002a640 addr v1:172.21.15.170:6790/0 2024-03-29T11:22:44.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.809+0000 7fd167fff640 10 monclient(hunting): picked mon.noname-c con 0x7fd16002b370 addr v1:172.21.15.184:6789/0 2024-03-29T11:22:44.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.809+0000 7fd167fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16002a120 con 0x7fd160006410 2024-03-29T11:22:44.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.809+0000 7fd167fff640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1600c75b0 con 0x7fd16002b370 2024-03-29T11:22:44.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.810+0000 7fd167fff640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd160033110 con 0x7fd16002a640 2024-03-29T11:22:44.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.810+0000 7fd167fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:44.787 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.810+0000 7fd167fff640 10 monclient(hunting): authenticate will time out at 1546.121033s 2024-03-29T11:22:44.789 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.812+0000 7fd1667fc640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fd16002a640 0x7fd1600c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:49182/0 (socket says 172.21.15.170:49182) 2024-03-29T11:22:44.789 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.812+0000 7fd1667fc640 1 -- 172.21.15.170:0/3991228068 learned_addr learned my addr 172.21.15.170:0/3991228068 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:22:44.789 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.813+0000 7fd165ffb640 1 -- 172.21.15.170:0/3991228068 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918886237 0 0) 0x7fd1600c75b0 con 0x7fd16002b370 2024-03-29T11:22:44.790 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.813+0000 7fd165ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:44.790 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.813+0000 7fd165ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:44.790 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.813+0000 7fd165ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:44.790 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.813+0000 7fd165ffb640 10 monclient(hunting): my global_id is 4576 2024-03-29T11:22:44.790 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.813+0000 7fd165ffb640 1 -- 172.21.15.170:0/3991228068 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd150003520 con 0x7fd16002b370 2024-03-29T11:22:44.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 1 -- 172.21.15.170:0/3991228068 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2962041498 0 0) 0x7fd160033110 con 0x7fd16002a640 2024-03-29T11:22:44.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:44.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:44.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:44.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 10 monclient(hunting): my global_id is 4679 2024-03-29T11:22:44.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 1 -- 172.21.15.170:0/3991228068 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1600c75b0 con 0x7fd16002a640 2024-03-29T11:22:44.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 1 -- 172.21.15.170:0/3991228068 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 549382446 0 0) 0x7fd150003520 con 0x7fd16002b370 2024-03-29T11:22:44.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 1 -- 172.21.15.170:0/3991228068 >> v1:172.21.15.170:6790/0 conn(0x7fd16002a640 legacy=0x7fd1600c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:44.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 1 -- 172.21.15.170:0/3991228068 >> v1:172.21.15.170:6789/0 conn(0x7fd160006410 legacy=0x7fd1600345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:44.794 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:44.795 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 1 monclient: found mon.noname-c 2024-03-29T11:22:44.795 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:44.795 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.184:6789/0 2024-03-29T11:22:44.795 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 1 -- 172.21.15.170:0/3991228068 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1600332a0 con 0x7fd16002b370 2024-03-29T11:22:44.795 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:44.795 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:44.795 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:44.795 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 1 -- 172.21.15.170:0/3991228068 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fd158001690 con 0x7fd16002b370 2024-03-29T11:22:44.795 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:44.795 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:44.795 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd165ffb640 10 monclient: dump: 2024-03-29T11:22:44.795 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:44.795 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:44.795 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:44.795 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:44.795 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:44.796 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:44.796 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:44.796 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:44.796 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:44.796 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:44.796 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd167fff640 5 monclient: authenticate success, global_id 4576 2024-03-29T11:22:44.796 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.816+0000 7fd167fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:44.796 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.817+0000 7fd165ffb640 1 -- 172.21.15.170:0/3991228068 <== mon.1 v1:172.21.15.184:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1580017f0 con 0x7fd16002b370 2024-03-29T11:22:44.796 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.817+0000 7fd165ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:44.796 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.817+0000 7fd167fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:44.796 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.817+0000 7fd167fff640 4 set_mon_vals no callback set 2024-03-29T11:22:44.796 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.817+0000 7fd165ffb640 1 -- 172.21.15.170:0/3991228068 <== mon.1 v1:172.21.15.184:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fd158001b80 con 0x7fd16002b370 2024-03-29T11:22:44.796 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.817+0000 7fd165ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:44.796 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.817+0000 7fd165ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:44.797 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.817+0000 7fd165ffb640 10 monclient: dump: 2024-03-29T11:22:44.797 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:44.797 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:44.797 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:44.797 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:44.797 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:44.797 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:44.797 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:44.797 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:44.797 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:44.797 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:44.798 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.818+0000 7fd167fff640 10 monclient: shutdown 2024-03-29T11:22:44.798 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.818+0000 7fd167fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:44.798 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.818+0000 7fd167fff640 1 -- 172.21.15.170:0/3991228068 >> v1:172.21.15.184:6789/0 conn(0x7fd16002b370 legacy=0x7fd1600320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:44.798 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.818+0000 7fd167fff640 1 -- 172.21.15.170:0/3991228068 shutdown_connections 2024-03-29T11:22:44.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.818+0000 7fd167fff640 1 -- 172.21.15.170:0/3991228068 >> 172.21.15.170:0/3991228068 conn(0x7fd1600058f0 msgr2=0x7fd160005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:44.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.818+0000 7fd167fff640 1 -- 172.21.15.170:0/3991228068 shutdown_connections 2024-03-29T11:22:44.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.818+0000 7fd167fff640 1 -- 172.21.15.170:0/3991228068 wait complete. 2024-03-29T11:22:44.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.819+0000 7fd167fff640 10 monclient: build_initial_monmap 2024-03-29T11:22:44.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.819+0000 7fd167fff640 10 monclient: monmap: 2024-03-29T11:22:44.799 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:22:44.799 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:44.799 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:22:44.799 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:22:44.799 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:44.799 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:44.799 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:22:44.799 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:22:44.799 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:22:44.799 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:44.799 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.819+0000 7fd167fff640 1 librados: starting msgr at 2024-03-29T11:22:44.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.819+0000 7fd167fff640 1 librados: starting objecter 2024-03-29T11:22:44.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.820+0000 7fd167fff640 1 Processor -- start 2024-03-29T11:22:44.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.820+0000 7fd167fff640 1 -- start start 2024-03-29T11:22:44.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.820+0000 7fd167fff640 1 librados: setting wanted keys 2024-03-29T11:22:44.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.820+0000 7fd167fff640 1 librados: calling monclient init 2024-03-29T11:22:44.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.820+0000 7fd167fff640 10 monclient: init 2024-03-29T11:22:44.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.821+0000 7fd167fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:44.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.821+0000 7fd167fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:44.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.821+0000 7fd167fff640 10 monclient(hunting): picked mon.noname-c con 0x7fd160006410 addr v1:172.21.15.170:6790/0 2024-03-29T11:22:44.800 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.821+0000 7fd167fff640 10 monclient(hunting): picked mon.noname-b con 0x7fd16002a640 addr v1:172.21.15.184:6789/0 2024-03-29T11:22:44.801 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.823+0000 7fd167fff640 10 monclient(hunting): picked mon.noname-a con 0x7fd16002b370 addr v1:172.21.15.170:6789/0 2024-03-29T11:22:44.801 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.823+0000 7fd167fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1600d3970 con 0x7fd16002b370 2024-03-29T11:22:44.801 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.823+0000 7fd167fff640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1600d5750 con 0x7fd16002a640 2024-03-29T11:22:44.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.823+0000 7fd167fff640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1600d4380 con 0x7fd160006410 2024-03-29T11:22:44.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.823+0000 7fd167fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:44.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.823+0000 7fd167fff640 10 monclient(hunting): authenticate will time out at 1546.134736s 2024-03-29T11:22:44.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.823+0000 7fd166ffd640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7fd160006410 0x7fd1600d1830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:52294/0 (socket says 172.21.15.170:52294) 2024-03-29T11:22:44.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.823+0000 7fd166ffd640 1 -- 172.21.15.170:0/2689892276 learned_addr learned my addr 172.21.15.170:0/2689892276 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:22:44.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.824+0000 7fd14f7fe640 10 client.?.objecter ms_handle_connect 0x7fd160006410 2024-03-29T11:22:44.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.824+0000 7fd14f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:44.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.824+0000 7fd14f7fe640 1 -- 172.21.15.170:0/2689892276 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457337364 0 0) 0x7fd1600d4380 con 0x7fd160006410 2024-03-29T11:22:44.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.824+0000 7fd14f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:44.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.824+0000 7fd14f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:44.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.824+0000 7fd14f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:44.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.824+0000 7fd14f7fe640 10 monclient(hunting): my global_id is 4682 2024-03-29T11:22:44.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.824+0000 7fd14f7fe640 1 -- 172.21.15.170:0/2689892276 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd154004eb0 con 0x7fd160006410 2024-03-29T11:22:44.802 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.825+0000 7fd14f7fe640 1 -- 172.21.15.170:0/2689892276 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2234152450 0 0) 0x7fd154004eb0 con 0x7fd160006410 2024-03-29T11:22:44.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.825+0000 7fd14f7fe640 1 -- 172.21.15.170:0/2689892276 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1600d4380 con 0x7fd160006410 2024-03-29T11:22:44.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.825+0000 7fd14f7fe640 1 -- 172.21.15.170:0/2689892276 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fd16804fdd0 con 0x7fd160006410 2024-03-29T11:22:44.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.825+0000 7fd14f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:44.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.825+0000 7fd14f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:44.803 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.825+0000 7fd14f7fe640 10 monclient(hunting): dump: 2024-03-29T11:22:44.803 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:44.803 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:44.803 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:44.803 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:44.803 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:44.803 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:44.803 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:44.803 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:44.803 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:44.803 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:44.804 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.825+0000 7fd14f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:44.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.825+0000 7fd14f7fe640 1 -- 172.21.15.170:0/2689892276 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3868776146 0 0) 0x7fd1600d4380 con 0x7fd160006410 2024-03-29T11:22:44.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.825+0000 7fd14f7fe640 1 -- 172.21.15.170:0/2689892276 >> v1:172.21.15.184:6789/0 conn(0x7fd16002a640 legacy=0x7fd1600d5040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:44.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.826+0000 7fd14f7fe640 1 -- 172.21.15.170:0/2689892276 >> v1:172.21.15.170:6789/0 conn(0x7fd16002b370 legacy=0x7fd16016b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:44.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.826+0000 7fd14f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:44.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.826+0000 7fd14f7fe640 1 monclient: found mon.c 2024-03-29T11:22:44.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.826+0000 7fd14f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:44.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.826+0000 7fd14f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:44.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.826+0000 7fd14f7fe640 1 -- 172.21.15.170:0/2689892276 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1600d4660 con 0x7fd160006410 2024-03-29T11:22:44.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.826+0000 7fd14f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:44.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.826+0000 7fd14f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:44.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.826+0000 7fd14f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:44.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.826+0000 7fd167fff640 5 monclient: authenticate success, global_id 4682 2024-03-29T11:22:44.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.826+0000 7fd167fff640 10 monclient: _renew_subs 2024-03-29T11:22:44.805 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.826+0000 7fd167fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:44.806 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.826+0000 7fd167fff640 1 -- 172.21.15.170:0/2689892276 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1600d4940 con 0x7fd160006410 2024-03-29T11:22:44.806 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.826+0000 7fd167fff640 10 client.4682.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:44.806 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.826+0000 7fd167fff640 10 monclient: _renew_subs 2024-03-29T11:22:44.806 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.826+0000 7fd167fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:44.806 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.826+0000 7fd167fff640 1 -- 172.21.15.170:0/2689892276 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1600d3570 con 0x7fd160006410 2024-03-29T11:22:44.806 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.826+0000 7fd167fff640 1 librados: init done 2024-03-29T11:22:44.806 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.828+0000 7fd167fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:44.806 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.828+0000 7fd167fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:44.806 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.828+0000 7fd167fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:44.806 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.828+0000 7fd167fff640 1 -- 172.21.15.170:0/2689892276 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd160004cd0 con 0x7fd160006410 2024-03-29T11:22:44.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.829+0000 7fd14f7fe640 1 -- 172.21.15.170:0/2689892276 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd16804f400 con 0x7fd160006410 2024-03-29T11:22:44.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.829+0000 7fd14f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:44.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.829+0000 7fd14f7fe640 1 -- 172.21.15.170:0/2689892276 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fd168075f10 con 0x7fd160006410 2024-03-29T11:22:44.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.829+0000 7fd14f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:44.808 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.829+0000 7fd14f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:44.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.829+0000 7fd14f7fe640 10 monclient: dump: 2024-03-29T11:22:44.809 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:44.809 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:44.809 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:44.809 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:44.809 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:44.809 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:44.809 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:44.809 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:44.809 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:44.809 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:44.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.829+0000 7fd1657fa640 4 set_mon_vals no callback set 2024-03-29T11:22:44.809 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.831+0000 7fd14f7fe640 1 -- 172.21.15.170:0/2689892276 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fd168067bf0 con 0x7fd160006410 2024-03-29T11:22:44.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.834+0000 7fd14f7fe640 1 -- 172.21.15.170:0/2689892276 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4038649127 0 0) 0x7fd1680ff670 con 0x7fd160006410 2024-03-29T11:22:44.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.834+0000 7fd14f7fe640 10 client.4682.objecter ms_dispatch 0x7fd168000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:22:44.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.834+0000 7fd14f7fe640 3 client.4682.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:22:44.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.834+0000 7fd14f7fe640 3 client.4682.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:22:44.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.834+0000 7fd14f7fe640 20 client.4682.objecter dump_active .. 0 homeless 2024-03-29T11:22:44.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.837+0000 7fd1667fc640 10 monclient: get_auth_request con 0x7fd15406dad0 auth_method 0 2024-03-29T11:22:44.814 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.837+0000 7fd14f7fe640 10 client.4682.objecter ms_handle_connect 0x7fd15406dad0 2024-03-29T11:22:44.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.839+0000 7fd14f7fe640 1 -- 172.21.15.170:0/2689892276 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd1680cb0f0 con 0x7fd160006410 2024-03-29T11:22:44.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.839+0000 7fd14f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:44.817 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:44.839+0000 7fd14f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:45.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.217+0000 7fd167fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:45.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.217+0000 7fd167fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:45.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.217+0000 7fd167fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:45.194 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.217+0000 7fd167fff640 1 -- 172.21.15.170:0/2689892276 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1600d4ad0 con 0x7fd160006410 2024-03-29T11:22:45.195 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.218+0000 7fd14f7fe640 1 -- 172.21.15.170:0/2689892276 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3062895256) 0x7fd1680cd020 con 0x7fd160006410 2024-03-29T11:22:45.195 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.218+0000 7fd14f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:45.195 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.218+0000 7fd14f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:45.195 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:22:45.195 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":11,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:22:06.930367+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:22:06.930367+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:45.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.222+0000 7fd167fff640 10 librados: watch_flush enter 2024-03-29T11:22:45.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.222+0000 7fd167fff640 10 librados: watch_flush exit 2024-03-29T11:22:45.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.222+0000 7fd167fff640 20 client.4682.objecter shutdown clearing up homeless session... 2024-03-29T11:22:45.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.222+0000 7fd167fff640 10 client.4682.objecter successfully canceled tick 2024-03-29T11:22:45.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.222+0000 7fd167fff640 1 -- 172.21.15.170:0/2689892276 >> v1:172.21.15.170:6816/3298994444 conn(0x7fd15406dad0 legacy=0x7fd15406ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:45.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.222+0000 7fd167fff640 10 monclient: shutdown 2024-03-29T11:22:45.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.222+0000 7fd167fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:45.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.222+0000 7fd167fff640 1 -- 172.21.15.170:0/2689892276 >> v1:172.21.15.170:6790/0 conn(0x7fd160006410 legacy=0x7fd1600d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:45.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.222+0000 7fd167fff640 1 -- 172.21.15.170:0/2689892276 shutdown_connections 2024-03-29T11:22:45.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.222+0000 7fd167fff640 1 -- 172.21.15.170:0/2689892276 >> 172.21.15.170:0/2689892276 conn(0x7fd1600058f0 msgr2=0x7fd1600043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:45.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.223+0000 7fd167fff640 1 -- 172.21.15.170:0/2689892276 shutdown_connections 2024-03-29T11:22:45.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.223+0000 7fd167fff640 1 -- 172.21.15.170:0/2689892276 wait complete. 2024-03-29T11:22:45.200 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:45.223+0000 7fd167fff640 1 librados: shutdown 2024-03-29T11:22:45.215 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:52.022 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:52.174 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.197+0000 7fdfddcba640 10 monclient: get_monmap_and_config 2024-03-29T11:22:52.174 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.197+0000 7fdfddcba640 10 monclient: build_initial_monmap 2024-03-29T11:22:52.174 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.197+0000 7fdfddcba640 10 monclient: monmap: 2024-03-29T11:22:52.174 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:22:52.174 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:52.174 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:22:52.197489+0000 2024-03-29T11:22:52.174 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:22:52.197489+0000 2024-03-29T11:22:52.175 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:52.175 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:52.175 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:22:52.175 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:22:52.175 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:22:52.175 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:52.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.198+0000 7fdfddcba640 1 Processor -- start 2024-03-29T11:22:52.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.198+0000 7fdfddcba640 1 -- start start 2024-03-29T11:22:52.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.198+0000 7fdfddcba640 10 monclient: init 2024-03-29T11:22:52.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.199+0000 7fdfddcba640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:52.176 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.199+0000 7fdfddcba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:52.179 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.201+0000 7fdfddcba640 10 monclient(hunting): picked mon.noname-b con 0x7fdfd0006a70 addr v1:172.21.15.170:6790/0 2024-03-29T11:22:52.179 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.201+0000 7fdfddcba640 10 monclient(hunting): picked mon.noname-c con 0x7fdfd002b100 addr v1:172.21.15.184:6789/0 2024-03-29T11:22:52.182 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.202+0000 7fdfddcba640 10 monclient(hunting): picked mon.noname-a con 0x7fdfd0029c30 addr v1:172.21.15.170:6789/0 2024-03-29T11:22:52.182 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.202+0000 7fdfddcba640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd002a270 con 0x7fdfd0029c30 2024-03-29T11:22:52.182 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.202+0000 7fdfddcba640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd002c440 con 0x7fdfd002b100 2024-03-29T11:22:52.182 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.202+0000 7fdfddcba640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd00c9440 con 0x7fdfd0006a70 2024-03-29T11:22:52.182 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.202+0000 7fdfddcba640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:52.182 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.202+0000 7fdfddcba640 10 monclient(hunting): authenticate will time out at 1553.513886s 2024-03-29T11:22:52.183 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.206+0000 7fdfd7fff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fdfd0029c30 0x7fdfd00d74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:43149/0 (socket says 172.21.15.170:43149) 2024-03-29T11:22:52.183 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.206+0000 7fdfd7fff640 1 -- 172.21.15.170:0/774805482 learned_addr learned my addr 172.21.15.170:0/774805482 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:22:52.186 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.209+0000 7fdfd67fc640 1 -- 172.21.15.170:0/774805482 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2613158668 0 0) 0x7fdfd00c9440 con 0x7fdfd0006a70 2024-03-29T11:22:52.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.209+0000 7fdfd67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:52.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.209+0000 7fdfd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:52.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.209+0000 7fdfd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:52.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.209+0000 7fdfd67fc640 10 monclient(hunting): my global_id is 4685 2024-03-29T11:22:52.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.209+0000 7fdfd67fc640 1 -- 172.21.15.170:0/774805482 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfc8003520 con 0x7fdfd0006a70 2024-03-29T11:22:52.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.209+0000 7fdfd67fc640 1 -- 172.21.15.170:0/774805482 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1366267437 0 0) 0x7fdfc8003520 con 0x7fdfd0006a70 2024-03-29T11:22:52.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.209+0000 7fdfd67fc640 1 -- 172.21.15.170:0/774805482 >> v1:172.21.15.184:6789/0 conn(0x7fdfd002b100 legacy=0x7fdfd00cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:52.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.209+0000 7fdfd67fc640 1 -- 172.21.15.170:0/774805482 >> v1:172.21.15.170:6789/0 conn(0x7fdfd0029c30 legacy=0x7fdfd00d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:52.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.209+0000 7fdfd67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:52.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.209+0000 7fdfd67fc640 1 monclient: found mon.noname-b 2024-03-29T11:22:52.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.209+0000 7fdfd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:52.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.209+0000 7fdfd67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:22:52.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.209+0000 7fdfd67fc640 1 -- 172.21.15.170:0/774805482 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd00c95d0 con 0x7fdfd0006a70 2024-03-29T11:22:52.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.209+0000 7fdfd67fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:52.187 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.209+0000 7fdfd67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:52.188 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.209+0000 7fdfd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:52.188 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.209+0000 7fdfd67fc640 1 -- 172.21.15.170:0/774805482 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fdfc4001690 con 0x7fdfd0006a70 2024-03-29T11:22:52.188 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.209+0000 7fdfd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:52.188 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.210+0000 7fdfd67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:52.188 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.210+0000 7fdfd67fc640 10 monclient: dump: 2024-03-29T11:22:52.188 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:52.188 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:52.188 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:52.188 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:52.188 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:52.188 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:52.188 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:52.188 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:52.188 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:52.188 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:52.190 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.210+0000 7fdfd67fc640 1 -- 172.21.15.170:0/774805482 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfc4001860 con 0x7fdfd0006a70 2024-03-29T11:22:52.190 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.210+0000 7fdfd67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:52.190 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.210+0000 7fdfd67fc640 1 -- 172.21.15.170:0/774805482 <== mon.2 v1:172.21.15.170:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fdfc4001c70 con 0x7fdfd0006a70 2024-03-29T11:22:52.190 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.210+0000 7fdfd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:52.190 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.210+0000 7fdfd67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:52.190 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.210+0000 7fdfd67fc640 10 monclient: dump: 2024-03-29T11:22:52.190 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:52.190 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:52.190 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:52.190 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:52.190 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:52.190 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:52.190 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:52.190 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:52.191 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:52.191 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:52.191 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.211+0000 7fdfddcba640 5 monclient: authenticate success, global_id 4685 2024-03-29T11:22:52.191 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.211+0000 7fdfddcba640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:52.191 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.211+0000 7fdfddcba640 4 set_mon_vals no callback set 2024-03-29T11:22:52.191 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.212+0000 7fdfddcba640 10 monclient: shutdown 2024-03-29T11:22:52.191 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.212+0000 7fdfddcba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:52.191 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.212+0000 7fdfddcba640 1 -- 172.21.15.170:0/774805482 >> v1:172.21.15.170:6790/0 conn(0x7fdfd0006a70 legacy=0x7fdfd00c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:52.191 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.213+0000 7fdfddcba640 1 -- 172.21.15.170:0/774805482 shutdown_connections 2024-03-29T11:22:52.191 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.213+0000 7fdfddcba640 1 -- 172.21.15.170:0/774805482 >> 172.21.15.170:0/774805482 conn(0x7fdfd00058f0 msgr2=0x7fdfd0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:52.195 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.215+0000 7fdfddcba640 1 -- 172.21.15.170:0/774805482 shutdown_connections 2024-03-29T11:22:52.195 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.218+0000 7fdfddcba640 1 -- 172.21.15.170:0/774805482 wait complete. 2024-03-29T11:22:52.197 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.219+0000 7fdfddcba640 10 monclient: build_initial_monmap 2024-03-29T11:22:52.197 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.219+0000 7fdfddcba640 10 monclient: monmap: 2024-03-29T11:22:52.197 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:22:52.197 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:52.197 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:22:52.197 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:22:52.197 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:52.197 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:52.197 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:22:52.198 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:22:52.198 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:22:52.198 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:52.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.220+0000 7fdfddcba640 1 librados: starting msgr at 2024-03-29T11:22:52.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.220+0000 7fdfddcba640 1 librados: starting objecter 2024-03-29T11:22:52.198 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.220+0000 7fdfddcba640 1 Processor -- start 2024-03-29T11:22:52.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.221+0000 7fdfddcba640 1 -- start start 2024-03-29T11:22:52.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.221+0000 7fdfddcba640 1 librados: setting wanted keys 2024-03-29T11:22:52.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.221+0000 7fdfddcba640 1 librados: calling monclient init 2024-03-29T11:22:52.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.221+0000 7fdfddcba640 10 monclient: init 2024-03-29T11:22:52.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.221+0000 7fdfddcba640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:52.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.221+0000 7fdfddcba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:52.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.221+0000 7fdfddcba640 10 monclient(hunting): picked mon.noname-a con 0x7fdfd0006a70 addr v1:172.21.15.170:6789/0 2024-03-29T11:22:52.199 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.221+0000 7fdfddcba640 10 monclient(hunting): picked mon.noname-b con 0x7fdfd0029c30 addr v1:172.21.15.184:6789/0 2024-03-29T11:22:52.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.223+0000 7fdfddcba640 10 monclient(hunting): picked mon.noname-c con 0x7fdfd0166cd0 addr v1:172.21.15.170:6790/0 2024-03-29T11:22:52.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.223+0000 7fdfddcba640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd016cd70 con 0x7fdfd0006a70 2024-03-29T11:22:52.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.223+0000 7fdfddcba640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd00caad0 con 0x7fdfd0029c30 2024-03-29T11:22:52.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.223+0000 7fdfddcba640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd002f3f0 con 0x7fdfd0166cd0 2024-03-29T11:22:52.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.223+0000 7fdfddcba640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:52.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.223+0000 7fdfddcba640 10 monclient(hunting): authenticate will time out at 1553.534358s 2024-03-29T11:22:52.203 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.225+0000 7fdfd77fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fdfd0006a70 0x7fdfd002db80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:43501/0 (socket says 172.21.15.170:43501) 2024-03-29T11:22:52.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.225+0000 7fdfd77fe640 1 -- 172.21.15.170:0/759945873 learned_addr learned my addr 172.21.15.170:0/759945873 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:22:52.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.226+0000 7fdfc3fff640 10 client.?.objecter ms_handle_connect 0x7fdfd0006a70 2024-03-29T11:22:52.204 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.226+0000 7fdfc3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:52.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 client.?.objecter ms_handle_connect 0x7fdfd0166cd0 2024-03-29T11:22:52.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:52.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 client.?.objecter ms_handle_connect 0x7fdfd0029c30 2024-03-29T11:22:52.207 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:52.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 1 -- 172.21.15.170:0/759945873 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327379888 0 0) 0x7fdfd002f3f0 con 0x7fdfd0166cd0 2024-03-29T11:22:52.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:52.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:52.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:52.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 monclient(hunting): my global_id is 4688 2024-03-29T11:22:52.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 1 -- 172.21.15.170:0/759945873 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfbc004eb0 con 0x7fdfd0166cd0 2024-03-29T11:22:52.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 1 -- 172.21.15.170:0/759945873 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390152826 0 0) 0x7fdfd016cd70 con 0x7fdfd0006a70 2024-03-29T11:22:52.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:52.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:52.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:52.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 monclient(hunting): my global_id is 4602 2024-03-29T11:22:52.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 1 -- 172.21.15.170:0/759945873 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfd002f3f0 con 0x7fdfd0006a70 2024-03-29T11:22:52.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 1 -- 172.21.15.170:0/759945873 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497755846 0 0) 0x7fdfd00caad0 con 0x7fdfd0029c30 2024-03-29T11:22:52.208 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:52.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:52.209 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:52.210 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 monclient(hunting): my global_id is 4582 2024-03-29T11:22:52.210 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 1 -- 172.21.15.170:0/759945873 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfd016cd70 con 0x7fdfd0029c30 2024-03-29T11:22:52.210 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 1 -- 172.21.15.170:0/759945873 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1089438699 0 0) 0x7fdfbc004eb0 con 0x7fdfd0166cd0 2024-03-29T11:22:52.210 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 1 -- 172.21.15.170:0/759945873 --> v1:172.21.15.170:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfd00caad0 con 0x7fdfd0166cd0 2024-03-29T11:22:52.210 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 1 -- 172.21.15.170:0/759945873 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fdfd804fdd0 con 0x7fdfd0166cd0 2024-03-29T11:22:52.210 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:52.210 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:52.210 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 monclient(hunting): dump: 2024-03-29T11:22:52.210 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:52.210 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:52.210 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:52.211 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:52.211 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:52.211 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:52.211 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:52.211 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:52.211 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:52.211 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:52.211 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.229+0000 7fdfc3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:52.211 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfc3fff640 1 -- 172.21.15.170:0/759945873 <== mon.2 v1:172.21.15.170:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1855046841 0 0) 0x7fdfd00caad0 con 0x7fdfd0166cd0 2024-03-29T11:22:52.211 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfc3fff640 1 -- 172.21.15.170:0/759945873 >> v1:172.21.15.184:6789/0 conn(0x7fdfd0029c30 legacy=0x7fdfd002cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:52.211 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfc3fff640 1 -- 172.21.15.170:0/759945873 >> v1:172.21.15.170:6789/0 conn(0x7fdfd0006a70 legacy=0x7fdfd002db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:52.211 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfc3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:52.211 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfc3fff640 1 monclient: found mon.c 2024-03-29T11:22:52.211 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:52.211 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfc3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:52.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfc3fff640 1 -- 172.21.15.170:0/759945873 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd00cc8a0 con 0x7fdfd0166cd0 2024-03-29T11:22:52.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfc3fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:52.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfc3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:52.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:52.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfddcba640 5 monclient: authenticate success, global_id 4688 2024-03-29T11:22:52.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfddcba640 10 monclient: _renew_subs 2024-03-29T11:22:52.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfddcba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:52.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfddcba640 1 -- 172.21.15.170:0/759945873 --> v1:172.21.15.170:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd016d910 con 0x7fdfd0166cd0 2024-03-29T11:22:52.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfddcba640 10 client.4688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:52.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfddcba640 10 monclient: _renew_subs 2024-03-29T11:22:52.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfddcba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:52.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfddcba640 1 -- 172.21.15.170:0/759945873 --> v1:172.21.15.170:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd0167120 con 0x7fdfd0166cd0 2024-03-29T11:22:52.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfddcba640 1 librados: init done 2024-03-29T11:22:52.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfc3fff640 1 -- 172.21.15.170:0/759945873 <== mon.2 v1:172.21.15.170:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfd804f400 con 0x7fdfd0166cd0 2024-03-29T11:22:52.212 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfc3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:52.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfc3fff640 1 -- 172.21.15.170:0/759945873 <== mon.2 v1:172.21.15.170:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7fdfd8075f10 con 0x7fdfd0166cd0 2024-03-29T11:22:52.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:52.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfc3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:52.213 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfc3fff640 10 monclient: dump: 2024-03-29T11:22:52.213 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:52.213 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:52.213 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:52.213 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:52.213 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:52.213 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:52.213 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:52.213 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:52.213 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:52.213 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:52.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.230+0000 7fdfd5ffb640 4 set_mon_vals no callback set 2024-03-29T11:22:52.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.232+0000 7fdfddcba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:52.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.232+0000 7fdfddcba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:52.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.232+0000 7fdfddcba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:52.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.232+0000 7fdfddcba640 1 -- 172.21.15.170:0/759945873 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb8005080 con 0x7fdfd0166cd0 2024-03-29T11:22:52.214 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.232+0000 7fdfc3fff640 1 -- 172.21.15.170:0/759945873 <== mon.2 v1:172.21.15.170:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7fdfd8067bf0 con 0x7fdfd0166cd0 2024-03-29T11:22:52.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.233+0000 7fdfd77fe640 10 monclient: get_auth_request con 0x7fdfbc06dad0 auth_method 0 2024-03-29T11:22:52.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.234+0000 7fdfc3fff640 10 client.4688.objecter ms_handle_connect 0x7fdfbc06dad0 2024-03-29T11:22:52.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.234+0000 7fdfc3fff640 1 -- 172.21.15.170:0/759945873 <== mon.2 v1:172.21.15.170:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4038649127 0 0) 0x7fdfd80ff410 con 0x7fdfd0166cd0 2024-03-29T11:22:52.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.234+0000 7fdfc3fff640 10 client.4688.objecter ms_dispatch 0x7fdfd8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:22:52.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.234+0000 7fdfc3fff640 3 client.4688.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:22:52.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.234+0000 7fdfc3fff640 3 client.4688.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:22:52.215 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.234+0000 7fdfc3fff640 20 client.4688.objecter dump_active .. 0 homeless 2024-03-29T11:22:52.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.243+0000 7fdfc3fff640 1 -- 172.21.15.170:0/759945873 <== mon.2 v1:172.21.15.170:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdfd80cacf0 con 0x7fdfd0166cd0 2024-03-29T11:22:52.219 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.243+0000 7fdfc3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:52.220 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.243+0000 7fdfc3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:52.540 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.563+0000 7fdfddcba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:52.540 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.563+0000 7fdfddcba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:52.540 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.563+0000 7fdfddcba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.170:6790/0 2024-03-29T11:22:52.540 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.563+0000 7fdfddcba640 1 -- 172.21.15.170:0/759945873 --> v1:172.21.15.170:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfb8005330 con 0x7fdfd0166cd0 2024-03-29T11:22:52.543 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.566+0000 7fdfc3fff640 1 -- 172.21.15.170:0/759945873 <== mon.2 v1:172.21.15.170:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3062895256) 0x7fdfd80ccc20 con 0x7fdfd0166cd0 2024-03-29T11:22:52.543 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.566+0000 7fdfc3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:52.543 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.566+0000 7fdfc3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:52.544 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:22:52.544 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":11,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:22:06.930367+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:22:06.930367+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:52.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.570+0000 7fdfddcba640 10 librados: watch_flush enter 2024-03-29T11:22:52.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.570+0000 7fdfddcba640 10 librados: watch_flush exit 2024-03-29T11:22:52.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.570+0000 7fdfddcba640 20 client.4688.objecter shutdown clearing up homeless session... 2024-03-29T11:22:52.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.570+0000 7fdfddcba640 10 client.4688.objecter successfully canceled tick 2024-03-29T11:22:52.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.570+0000 7fdfddcba640 1 -- 172.21.15.170:0/759945873 >> v1:172.21.15.170:6816/3298994444 conn(0x7fdfbc06dad0 legacy=0x7fdfbc06ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:52.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.570+0000 7fdfddcba640 10 monclient: shutdown 2024-03-29T11:22:52.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.570+0000 7fdfddcba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:52.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.570+0000 7fdfddcba640 1 -- 172.21.15.170:0/759945873 >> v1:172.21.15.170:6790/0 conn(0x7fdfd0166cd0 legacy=0x7fdfd016d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:52.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.571+0000 7fdfddcba640 1 -- 172.21.15.170:0/759945873 shutdown_connections 2024-03-29T11:22:52.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.571+0000 7fdfddcba640 1 -- 172.21.15.170:0/759945873 >> 172.21.15.170:0/759945873 conn(0x7fdfd00058f0 msgr2=0x7fdfd00384c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:52.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.571+0000 7fdfddcba640 1 -- 172.21.15.170:0/759945873 shutdown_connections 2024-03-29T11:22:52.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.571+0000 7fdfddcba640 1 -- 172.21.15.170:0/759945873 wait complete. 2024-03-29T11:22:52.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:52.571+0000 7fdfddcba640 1 librados: shutdown 2024-03-29T11:22:52.565 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:59.371 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:59.520 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.543+0000 7f97f973b640 10 monclient: get_monmap_and_config 2024-03-29T11:22:59.521 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.543+0000 7f97f973b640 10 monclient: build_initial_monmap 2024-03-29T11:22:59.521 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.543+0000 7f97f973b640 10 monclient: monmap: 2024-03-29T11:22:59.521 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:22:59.521 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:59.521 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:22:59.543915+0000 2024-03-29T11:22:59.521 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:22:59.543915+0000 2024-03-29T11:22:59.521 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:59.521 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:59.521 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:22:59.521 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c 2024-03-29T11:22:59.521 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-b 2024-03-29T11:22:59.521 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:59.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.544+0000 7f97f973b640 1 Processor -- start 2024-03-29T11:22:59.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.544+0000 7f97f973b640 1 -- start start 2024-03-29T11:22:59.522 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.544+0000 7f97f973b640 10 monclient: init 2024-03-29T11:22:59.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.545+0000 7f97f973b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:59.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.545+0000 7f97f973b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:59.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.545+0000 7f97f973b640 10 monclient(hunting): picked mon.noname-c con 0x7f97ec006e30 addr v1:172.21.15.184:6789/0 2024-03-29T11:22:59.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.545+0000 7f97f973b640 10 monclient(hunting): picked mon.noname-a con 0x7f97ec035e00 addr v1:172.21.15.170:6789/0 2024-03-29T11:22:59.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.545+0000 7f97f973b640 10 monclient(hunting): picked mon.noname-b con 0x7f97ec0c9670 addr v1:172.21.15.170:6790/0 2024-03-29T11:22:59.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.545+0000 7f97f973b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97ec035970 con 0x7f97ec035e00 2024-03-29T11:22:59.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.545+0000 7f97f973b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97ec036280 con 0x7f97ec006e30 2024-03-29T11:22:59.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.545+0000 7f97f973b640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97ec035640 con 0x7f97ec0c9670 2024-03-29T11:22:59.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.545+0000 7f97f973b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:59.523 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.545+0000 7f97f973b640 10 monclient(hunting): authenticate will time out at 1560.856280s 2024-03-29T11:22:59.525 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.548+0000 7f97f37fe640 1 --1- >> v1:172.21.15.170:6790/0 conn(0x7f97ec0c9670 0x7f97ec0d7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6790/0 says I am v1:172.21.15.170:59261/0 (socket says 172.21.15.170:59261) 2024-03-29T11:22:59.525 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.548+0000 7f97f37fe640 1 -- 172.21.15.170:0/2125459941 learned_addr learned my addr 172.21.15.170:0/2125459941 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:22:59.526 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.548+0000 7f97f1ffb640 1 -- 172.21.15.170:0/2125459941 <== mon.2 v1:172.21.15.170:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294259004 0 0) 0x7f97ec035640 con 0x7f97ec0c9670 2024-03-29T11:22:59.526 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.548+0000 7f97f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:59.526 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.548+0000 7f97f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:59.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.548+0000 7f97f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:59.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.548+0000 7f97f1ffb640 10 monclient(hunting): my global_id is 4691 2024-03-29T11:22:59.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.548+0000 7f97f1ffb640 1 -- 172.21.15.170:0/2125459941 --> v1:172.21.15.170:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97e0003520 con 0x7f97ec0c9670 2024-03-29T11:22:59.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.548+0000 7f97f1ffb640 1 -- 172.21.15.170:0/2125459941 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914102910 0 0) 0x7f97ec035970 con 0x7f97ec035e00 2024-03-29T11:22:59.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.548+0000 7f97f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:59.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.548+0000 7f97f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:59.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.548+0000 7f97f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:59.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.548+0000 7f97f1ffb640 10 monclient(hunting): my global_id is 4605 2024-03-29T11:22:59.527 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.548+0000 7f97f1ffb640 1 -- 172.21.15.170:0/2125459941 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97ec035640 con 0x7f97ec035e00 2024-03-29T11:22:59.530 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.549+0000 7f97f1ffb640 1 -- 172.21.15.170:0/2125459941 <== mon.2 v1:172.21.15.170:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3185946802 0 0) 0x7f97e0003520 con 0x7f97ec0c9670 2024-03-29T11:22:59.530 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.549+0000 7f97f1ffb640 1 -- 172.21.15.170:0/2125459941 >> v1:172.21.15.184:6789/0 conn(0x7f97ec006e30 legacy=0x7f97ec0cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:59.530 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.549+0000 7f97f1ffb640 1 -- 172.21.15.170:0/2125459941 >> v1:172.21.15.170:6789/0 conn(0x7f97ec035e00 legacy=0x7f97ec0c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:59.530 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.550+0000 7f97f1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:59.530 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.550+0000 7f97f1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:22:59.530 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.550+0000 7f97f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:59.530 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.550+0000 7f97f1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.170:6790/0 2024-03-29T11:22:59.530 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.550+0000 7f97f1ffb640 1 -- 172.21.15.170:0/2125459941 --> v1:172.21.15.170:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec034e20 con 0x7f97ec0c9670 2024-03-29T11:22:59.531 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.550+0000 7f97f1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:59.531 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.550+0000 7f97f1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:59.531 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.550+0000 7f97f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:59.531 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.550+0000 7f97f1ffb640 1 -- 172.21.15.170:0/2125459941 <== mon.2 v1:172.21.15.170:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f97f404f730 con 0x7f97ec0c9670 2024-03-29T11:22:59.531 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.550+0000 7f97f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:59.531 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.550+0000 7f97f1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:59.531 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.550+0000 7f97f1ffb640 10 monclient: dump: 2024-03-29T11:22:59.531 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:59.531 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:59.531 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:59.531 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:59.531 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:59.531 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:59.531 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:59.531 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:59.532 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:59.532 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:59.532 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.550+0000 7f97f973b640 5 monclient: authenticate success, global_id 4691 2024-03-29T11:22:59.532 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.550+0000 7f97f973b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:59.532 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.551+0000 7f97f1ffb640 1 -- 172.21.15.170:0/2125459941 <== mon.2 v1:172.21.15.170:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97f4075f10 con 0x7f97ec0c9670 2024-03-29T11:22:59.532 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.551+0000 7f97f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:59.532 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.551+0000 7f97f973b640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:59.532 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.551+0000 7f97f973b640 4 set_mon_vals no callback set 2024-03-29T11:22:59.532 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.552+0000 7f97f973b640 10 monclient: shutdown 2024-03-29T11:22:59.532 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.552+0000 7f97f973b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:59.532 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.552+0000 7f97f973b640 1 -- 172.21.15.170:0/2125459941 >> v1:172.21.15.170:6790/0 conn(0x7f97ec0c9670 legacy=0x7f97ec0d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:59.532 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.553+0000 7f97f973b640 1 -- 172.21.15.170:0/2125459941 shutdown_connections 2024-03-29T11:22:59.532 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.553+0000 7f97f973b640 1 -- 172.21.15.170:0/2125459941 >> 172.21.15.170:0/2125459941 conn(0x7f97ec005c00 msgr2=0x7f97ec006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:59.532 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.553+0000 7f97f973b640 1 -- 172.21.15.170:0/2125459941 shutdown_connections 2024-03-29T11:22:59.540 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.554+0000 7f97f973b640 1 -- 172.21.15.170:0/2125459941 wait complete. 2024-03-29T11:22:59.540 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.555+0000 7f97f973b640 10 monclient: build_initial_monmap 2024-03-29T11:22:59.540 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.555+0000 7f97f973b640 10 monclient: monmap: 2024-03-29T11:22:59.540 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2024-03-29T11:22:59.540 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:59.540 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2024-03-29T11:22:59.540 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2024-03-29T11:22:59.540 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:59.540 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:59.541 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a 2024-03-29T11:22:59.541 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.noname-b 2024-03-29T11:22:59.541 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.noname-c 2024-03-29T11:22:59.541 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:59.541 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.556+0000 7f97f973b640 1 librados: starting msgr at 2024-03-29T11:22:59.541 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.556+0000 7f97f973b640 1 librados: starting objecter 2024-03-29T11:22:59.541 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.556+0000 7f97f973b640 1 Processor -- start 2024-03-29T11:22:59.541 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.556+0000 7f97f973b640 1 -- start start 2024-03-29T11:22:59.541 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.556+0000 7f97f973b640 1 librados: setting wanted keys 2024-03-29T11:22:59.541 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.556+0000 7f97f973b640 1 librados: calling monclient init 2024-03-29T11:22:59.541 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.556+0000 7f97f973b640 10 monclient: init 2024-03-29T11:22:59.541 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.557+0000 7f97f973b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:59.541 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.557+0000 7f97f973b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:59.541 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.557+0000 7f97f973b640 10 monclient(hunting): picked mon.noname-b con 0x7f97ec006e30 addr v1:172.21.15.184:6789/0 2024-03-29T11:22:59.541 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.557+0000 7f97f973b640 10 monclient(hunting): picked mon.noname-c con 0x7f97ec035e00 addr v1:172.21.15.170:6790/0 2024-03-29T11:22:59.542 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.557+0000 7f97f973b640 10 monclient(hunting): picked mon.noname-a con 0x7f97ec0c9670 addr v1:172.21.15.170:6789/0 2024-03-29T11:22:59.542 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.557+0000 7f97f973b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97ec0c2770 con 0x7f97ec0c9670 2024-03-29T11:22:59.542 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.557+0000 7f97f973b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97ec0cfab0 con 0x7f97ec006e30 2024-03-29T11:22:59.542 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.557+0000 7f97f973b640 1 -- --> v1:172.21.15.170:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97ec0cfd90 con 0x7f97ec035e00 2024-03-29T11:22:59.542 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.557+0000 7f97f973b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:59.542 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.557+0000 7f97f973b640 10 monclient(hunting): authenticate will time out at 1560.868797s 2024-03-29T11:22:59.542 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.557+0000 7f97f37fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f97ec0c9670 0x7f97ec1850e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:45798/0 (socket says 172.21.15.170:45798) 2024-03-29T11:22:59.542 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.557+0000 7f97f37fe640 1 -- 172.21.15.170:0/2095703729 learned_addr learned my addr 172.21.15.170:0/2095703729 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-29T11:22:59.542 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.557+0000 7f97eb7fe640 10 client.?.objecter ms_handle_connect 0x7f97ec0c9670 2024-03-29T11:22:59.546 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.557+0000 7f97eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:59.546 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.558+0000 7f97eb7fe640 1 -- 172.21.15.170:0/2095703729 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141195541 0 0) 0x7f97ec0c2770 con 0x7f97ec0c9670 2024-03-29T11:22:59.546 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.558+0000 7f97eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:59.546 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.558+0000 7f97eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:59.546 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.558+0000 7f97eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:59.546 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.558+0000 7f97eb7fe640 10 monclient(hunting): my global_id is 4608 2024-03-29T11:22:59.546 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.558+0000 7f97eb7fe640 1 -- 172.21.15.170:0/2095703729 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97d4004eb0 con 0x7f97ec0c9670 2024-03-29T11:22:59.546 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.560+0000 7f97eb7fe640 1 -- 172.21.15.170:0/2095703729 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3256720964 0 0) 0x7f97d4004eb0 con 0x7f97ec0c9670 2024-03-29T11:22:59.546 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.560+0000 7f97eb7fe640 1 -- 172.21.15.170:0/2095703729 --> v1:172.21.15.170:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97ec0c2770 con 0x7f97ec0c9670 2024-03-29T11:22:59.546 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.560+0000 7f97eb7fe640 1 -- 172.21.15.170:0/2095703729 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f97f404f730 con 0x7f97ec0c9670 2024-03-29T11:22:59.546 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.560+0000 7f97eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:59.546 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.560+0000 7f97eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:59.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.560+0000 7f97eb7fe640 10 monclient(hunting): dump: 2024-03-29T11:22:59.547 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:59.547 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:59.547 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:59.547 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:59.547 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:59.547 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:59.547 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:59.547 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:59.547 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:59.547 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:59.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.560+0000 7f97eb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:59.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.561+0000 7f97eb7fe640 1 -- 172.21.15.170:0/2095703729 <== mon.0 v1:172.21.15.170:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1603668729 0 0) 0x7f97ec0c2770 con 0x7f97ec0c9670 2024-03-29T11:22:59.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.561+0000 7f97eb7fe640 1 -- 172.21.15.170:0/2095703729 >> v1:172.21.15.170:6790/0 conn(0x7f97ec035e00 legacy=0x7f97ec032550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:59.547 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.561+0000 7f97eb7fe640 1 -- 172.21.15.170:0/2095703729 >> v1:172.21.15.184:6789/0 conn(0x7f97ec006e30 legacy=0x7f97ec0c1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:59.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.561+0000 7f97eb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:59.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.561+0000 7f97eb7fe640 1 monclient: found mon.a 2024-03-29T11:22:59.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.561+0000 7f97eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:59.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.561+0000 7f97eb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:22:59.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.561+0000 7f97eb7fe640 1 -- 172.21.15.170:0/2095703729 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec0cef00 con 0x7f97ec0c9670 2024-03-29T11:22:59.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.561+0000 7f97eb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:59.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.561+0000 7f97eb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:59.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.561+0000 7f97eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:59.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.562+0000 7f97f973b640 5 monclient: authenticate success, global_id 4608 2024-03-29T11:22:59.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.562+0000 7f97f973b640 10 monclient: _renew_subs 2024-03-29T11:22:59.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.562+0000 7f97f973b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:22:59.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.562+0000 7f97f973b640 1 -- 172.21.15.170:0/2095703729 --> v1:172.21.15.170:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ec0ccfc0 con 0x7f97ec0c9670 2024-03-29T11:22:59.548 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.562+0000 7f97f973b640 10 client.4608.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:59.549 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.562+0000 7f97f973b640 10 monclient: _renew_subs 2024-03-29T11:22:59.549 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.562+0000 7f97f973b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:22:59.549 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.562+0000 7f97f973b640 1 -- 172.21.15.170:0/2095703729 --> v1:172.21.15.170:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ec0c2de0 con 0x7f97ec0c9670 2024-03-29T11:22:59.549 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.562+0000 7f97f973b640 1 librados: init done 2024-03-29T11:22:59.549 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.562+0000 7f97eb7fe640 1 -- 172.21.15.170:0/2095703729 <== mon.0 v1:172.21.15.170:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97f404f400 con 0x7f97ec0c9670 2024-03-29T11:22:59.549 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.562+0000 7f97eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:59.551 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.562+0000 7f97f17fa640 4 set_mon_vals no callback set 2024-03-29T11:22:59.551 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.564+0000 7f97eb7fe640 1 -- 172.21.15.170:0/2095703729 <== mon.0 v1:172.21.15.170:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3136730759 0 0) 0x7f97f40758b0 con 0x7f97ec0c9670 2024-03-29T11:22:59.551 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.564+0000 7f97eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:59.551 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.564+0000 7f97eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:59.551 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.564+0000 7f97eb7fe640 10 monclient: dump: 2024-03-29T11:22:59.551 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2024-03-29T11:22:59.551 INFO:teuthology.misc.health.smithi170.stderr:fsid e9c3a847-ec94-40b5-8cc1-d1578b06cbfd 2024-03-29T11:22:59.551 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:59.551 INFO:teuthology.misc.health.smithi170.stderr:created 2024-03-29T11:11:25.655925+0000 2024-03-29T11:22:59.551 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:59.551 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2024-03-29T11:22:59.551 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.a 2024-03-29T11:22:59.551 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.184:6789/0 mon.b 2024-03-29T11:22:59.552 INFO:teuthology.misc.health.smithi170.stderr:2: v1:172.21.15.170:6790/0 mon.c 2024-03-29T11:22:59.552 INFO:teuthology.misc.health.smithi170.stderr: 2024-03-29T11:22:59.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.564+0000 7f97eb7fe640 1 -- 172.21.15.170:0/2095703729 <== mon.0 v1:172.21.15.170:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4097296763 0 0) 0x7f97f4067ea0 con 0x7f97ec0c9670 2024-03-29T11:22:59.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.565+0000 7f97f973b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:59.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.565+0000 7f97eb7fe640 1 -- 172.21.15.170:0/2095703729 <== mon.0 v1:172.21.15.170:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4038649127 0 0) 0x7f97f40ff700 con 0x7f97ec0c9670 2024-03-29T11:22:59.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.565+0000 7f97eb7fe640 10 client.4608.objecter ms_dispatch 0x7f97f4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:22:59.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.565+0000 7f97eb7fe640 3 client.4608.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:22:59.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.565+0000 7f97eb7fe640 3 client.4608.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:22:59.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.565+0000 7f97eb7fe640 20 client.4608.objecter dump_active .. 0 homeless 2024-03-29T11:22:59.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.566+0000 7f97f973b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:59.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.566+0000 7f97f973b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:22:59.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.566+0000 7f97f973b640 1 -- 172.21.15.170:0/2095703729 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d8005080 con 0x7f97ec0c9670 2024-03-29T11:22:59.552 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.569+0000 7f97f2ffd640 10 monclient: get_auth_request con 0x7f97d406dbf0 auth_method 0 2024-03-29T11:22:59.553 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.570+0000 7f97eb7fe640 10 client.4608.objecter ms_handle_connect 0x7f97d406dbf0 2024-03-29T11:22:59.554 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.577+0000 7f97eb7fe640 1 -- 172.21.15.170:0/2095703729 <== mon.0 v1:172.21.15.170:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f97f40cb060 con 0x7f97ec0c9670 2024-03-29T11:22:59.554 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.577+0000 7f97eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:59.554 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.577+0000 7f97eb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:59.882 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.904+0000 7f97f973b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:59.882 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.904+0000 7f97f973b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:59.882 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.904+0000 7f97f973b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.170:6789/0 2024-03-29T11:22:59.882 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.904+0000 7f97f973b640 1 -- 172.21.15.170:0/2095703729 --> v1:172.21.15.170:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97d8005330 con 0x7f97ec0c9670 2024-03-29T11:22:59.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.907+0000 7f97eb7fe640 1 -- 172.21.15.170:0/2095703729 <== mon.0 v1:172.21.15.170:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3062895256) 0x7f97f40ccf90 con 0x7f97ec0c9670 2024-03-29T11:22:59.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.907+0000 7f97eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:59.884 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.907+0000 7f97eb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:59.885 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-29T11:22:59.885 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":11,"fsid":"e9c3a847-ec94-40b5-8cc1-d1578b06cbfd","created":"2024-03-29T11:12:01.976172+0000","modified":"2024-03-29T11:22:06.930367+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:22:06.930367+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"42e2bbec-c0e5-45c1-a2c1-55829916ae2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"dcbc3485-c2c7-44a0-8cc5-648b65ebe62f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dbf4e71c-275e-4f4e-9f0e-dbdd355ac904","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"12e77d37-9ec0-49b7-acec-0a241a418fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"12a0b442-15be-4896-b121-042507f91881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9442ffc6-a927-47de-a9f3-7f8cb0ed40df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"29872cd7-f7bf-4bc0-b70f-e0ff589d69f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71f6869e-a468-4f2b-8f4b-c9c6005266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:59.888 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.911+0000 7f97f973b640 10 librados: watch_flush enter 2024-03-29T11:22:59.888 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.911+0000 7f97f973b640 10 librados: watch_flush exit 2024-03-29T11:22:59.888 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.911+0000 7f97f973b640 20 client.4608.objecter shutdown clearing up homeless session... 2024-03-29T11:22:59.889 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.911+0000 7f97f973b640 10 client.4608.objecter successfully canceled tick 2024-03-29T11:22:59.889 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.911+0000 7f97f973b640 1 -- 172.21.15.170:0/2095703729 >> v1:172.21.15.170:6816/3298994444 conn(0x7f97d406dbf0 legacy=0x7f97d40700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:59.889 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.911+0000 7f97f973b640 10 monclient: shutdown 2024-03-29T11:22:59.889 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.911+0000 7f97f973b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:59.889 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.911+0000 7f97f973b640 1 -- 172.21.15.170:0/2095703729 >> v1:172.21.15.170:6789/0 conn(0x7f97ec0c9670 legacy=0x7f97ec1850e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:59.889 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.911+0000 7f97f973b640 1 -- 172.21.15.170:0/2095703729 shutdown_connections 2024-03-29T11:22:59.889 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.911+0000 7f97f973b640 1 -- 172.21.15.170:0/2095703729 >> 172.21.15.170:0/2095703729 conn(0x7f97ec005c00 msgr2=0x7f97ec03c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:59.889 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.911+0000 7f97f973b640 1 -- 172.21.15.170:0/2095703729 shutdown_connections 2024-03-29T11:22:59.889 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.911+0000 7f97f973b640 1 -- 172.21.15.170:0/2095703729 wait complete. 2024-03-29T11:22:59.889 INFO:teuthology.misc.health.smithi170.stderr:2024-03-29T11:22:59.911+0000 7f97f973b640 1 librados: shutdown 2024-03-29T11:22:59.906 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:59.906 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:22:59.908 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:22:59.908 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:22:59.908 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:22:59.939 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:22:59.939 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:22:59.939 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:00.024 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:23:00.024 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:23:00.025 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:00.057 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:23:00.058 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:23:00.058 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:00.140 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:23:00.140 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:23:00.140 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:00.173 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:23:00.173 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:23:00.173 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:00.250 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:23:00.250 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:23:00.250 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:00.319 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:23:00.319 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:23:00.319 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:00.390 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:23:00.390 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:23:00.391 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:23:00.391 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:00.415 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:23:00.416 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:23:00.416 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:00.456 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:23:00.456 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:23:00.456 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:23:00.456 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:00.468 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:23:00.468 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:23:00.468 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:00.519 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:23:00.520 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:23:00.520 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:00.531 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:23:00.531 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:23:00.531 DEBUG:teuthology.orchestra.run.smithi170:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:23:00.636 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi170.front.sepia.ceph.com 2024-03-29T11:23:00.636 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:23:03.332 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi170.front.sepia.ceph.com 2024-03-29T11:23:03.333 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:23:03.442 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi170.front.sepia.ceph.com 2024-03-29T11:23:03.442 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:23:03.499 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi170.front.sepia.ceph.com 2024-03-29T11:23:03.499 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:23:03.592 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi184.front.sepia.ceph.com 2024-03-29T11:23:03.592 DEBUG:teuthology.orchestra.run.smithi184:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:23:07.059 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi184.front.sepia.ceph.com 2024-03-29T11:23:07.059 DEBUG:teuthology.orchestra.run.smithi184:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:23:07.175 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi184.front.sepia.ceph.com 2024-03-29T11:23:07.175 DEBUG:teuthology.orchestra.run.smithi184:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:23:07.233 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi184.front.sepia.ceph.com 2024-03-29T11:23:07.233 DEBUG:teuthology.orchestra.run.smithi184:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:23:07.294 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:23:07.296 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622678/data/mon.a.tgz 2024-03-29T11:23:07.296 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2024-03-29T11:23:07.314 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.bnaA95JXVh 2024-03-29T11:23:07.318 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.bnaA95JXVh 2024-03-29T11:23:07.422 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.bnaA95JXVh 2024-03-29T11:23:07.510 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.bnaA95JXVh is 126KB 2024-03-29T11:23:07.563 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.bnaA95JXVh 2024-03-29T11:23:07.577 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622678/data/mon.c.tgz 2024-03-29T11:23:07.578 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2024-03-29T11:23:07.632 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.UeJvDKWvSH 2024-03-29T11:23:07.632 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.UeJvDKWvSH 2024-03-29T11:23:07.720 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.UeJvDKWvSH 2024-03-29T11:23:07.804 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.UeJvDKWvSH is 126KB 2024-03-29T11:23:07.854 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.UeJvDKWvSH 2024-03-29T11:23:07.868 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622678/data/mon.b.tgz 2024-03-29T11:23:07.868 DEBUG:teuthology.orchestra.run.smithi184:> mktemp 2024-03-29T11:23:07.886 INFO:teuthology.orchestra.run.smithi184.stdout:/tmp/tmp.rYkpVsB0l0 2024-03-29T11:23:07.886 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.rYkpVsB0l0 2024-03-29T11:23:07.994 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0666 /tmp/tmp.rYkpVsB0l0 2024-03-29T11:23:08.081 DEBUG:teuthology.orchestra.remote:smithi184:/tmp/tmp.rYkpVsB0l0 is 126KB 2024-03-29T11:23:08.135 DEBUG:teuthology.orchestra.run.smithi184:> rm -fr /tmp/tmp.rYkpVsB0l0 2024-03-29T11:23:08.150 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:23:08.150 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:23:08.152 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:23:08.232 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:23:08.272 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:23:08.272 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:23:08.274 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622678/remote/smithi170/crash 2024-03-29T11:23:08.275 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:23:08.302 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622678/remote/smithi184/crash 2024-03-29T11:23:08.303 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:23:08.338 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:23:08.338 DEBUG:teuthology.orchestra.run.smithi170:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:23:08.345 DEBUG:teuthology.orchestra.run.smithi184:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:23:08.369 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40158.log 2024-03-29T11:23:08.369 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39534.log 2024-03-29T11:23:08.420 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39963.log 2024-03-29T11:23:08.420 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39534.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:23:08.420 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40158.log.gz 2024-03-29T11:23:08.420 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39768.log 2024-03-29T11:23:08.420 INFO:teuthology.orchestra.run.smithi170.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39534.log.gz 2024-03-29T11:23:08.420 INFO:teuthology.orchestra.run.smithi170.stderr:gzip/var/log/ceph/ceph-mon.a.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40197.log 2024-03-29T11:23:08.420 INFO:teuthology.orchestra.run.smithi170.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39963.log.gz 2024-03-29T11:23:08.420 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39339.log 2024-03-29T11:23:08.420 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39261.log 2024-03-29T11:23:08.421 INFO:teuthology.orchestra.run.smithi170.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39768.log.gz 2024-03-29T11:23:08.421 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38557.log 2024-03-29T11:23:08.421 INFO:teuthology.orchestra.run.smithi170.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40197.log.gz 2024-03-29T11:23:08.421 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41232.log 2024-03-29T11:23:08.421 INFO:teuthology.orchestra.run.smithi170.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39339.log.gz 2024-03-29T11:23:08.421 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.38557.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39261.log.gz 2024-03-29T11:23:08.421 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38674.log 2024-03-29T11:23:08.421 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41232.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28942.log 2024-03-29T11:23:08.421 INFO:teuthology.orchestra.run.smithi170.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38557.log.gz 2024-03-29T11:23:08.421 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.38674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38479.log 2024-03-29T11:23:08.421 INFO:teuthology.orchestra.run.smithi170.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41232.log.gz 2024-03-29T11:23:08.421 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph.tmp-client.admin.28942.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38674.log.gz 2024-03-29T11:23:08.421 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40002.log 2024-03-29T11:23:08.421 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28942.log.gz 2024-03-29T11:23:08.422 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.38479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39690.log 2024-03-29T11:23:08.422 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38754.log 2024-03-29T11:23:08.422 INFO:teuthology.orchestra.run.smithi170.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38479.log.gz 2024-03-29T11:23:08.422 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41895.log 2024-03-29T11:23:08.422 INFO:teuthology.orchestra.run.smithi170.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40002.log.gz 2024-03-29T11:23:08.422 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.38754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41700.log 2024-03-29T11:23:08.422 INFO:teuthology.orchestra.run.smithi170.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39690.log.gz 2024-03-29T11:23:08.422 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39417.log 2024-03-29T11:23:08.422 INFO:teuthology.orchestra.run.smithi170.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38754.log.gz 2024-03-29T11:23:08.422 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41700.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41895.log.gz 2024-03-29T11:23:08.422 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40449.log 2024-03-29T11:23:08.422 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39417.log: gzip -5 --verbose -- 85.6% /var/log/ceph/ceph-client.admin.39495.log 2024-03-29T11:23:08.422 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41700.log.gz 2024-03-29T11:23:08.423 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40996.log 2024-03-29T11:23:08.423 INFO:teuthology.orchestra.run.smithi170.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39417.log.gz 2024-03-29T11:23:08.423 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39495.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40449.log.gz 2024-03-29T11:23:08.423 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40839.log 2024-03-29T11:23:08.423 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41778.log 2024-03-29T11:23:08.423 INFO:teuthology.orchestra.run.smithi170.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39495.log.gz 2024-03-29T11:23:08.423 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41466.log 2024-03-29T11:23:08.423 INFO:teuthology.orchestra.run.smithi170.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40996.log.gz 2024-03-29T11:23:08.423 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41778.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40839.log.gz 2024-03-29T11:23:08.423 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38793.log 2024-03-29T11:23:08.423 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38832.log 2024-03-29T11:23:08.423 INFO:teuthology.orchestra.run.smithi170.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41778.log.gz 2024-03-29T11:23:08.423 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.38793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40332.log 2024-03-29T11:23:08.423 INFO:teuthology.orchestra.run.smithi170.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41466.log.gz 2024-03-29T11:23:08.423 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38793.log.gz 2024-03-29T11:23:08.423 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.38832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39612.log 2024-03-29T11:23:08.424 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41661.log 2024-03-29T11:23:08.424 INFO:teuthology.orchestra.run.smithi170.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38832.log.gz 2024-03-29T11:23:08.424 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39144.log 2024-03-29T11:23:08.424 INFO:teuthology.orchestra.run.smithi170.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40332.log.gz 2024-03-29T11:23:08.424 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41661.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39612.log.gz 2024-03-29T11:23:08.424 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41622.log 2024-03-29T11:23:08.424 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40722.log 2024-03-29T11:23:08.424 INFO:teuthology.orchestra.run.smithi170.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41661.log.gz 2024-03-29T11:23:08.424 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41505.log 2024-03-29T11:23:08.424 INFO:teuthology.orchestra.run.smithi170.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39144.log.gz 2024-03-29T11:23:08.424 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34749.log 2024-03-29T11:23:08.424 INFO:teuthology.orchestra.run.smithi170.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41622.log.gz 2024-03-29T11:23:08.424 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40918.log 2024-03-29T11:23:08.424 INFO:teuthology.orchestra.run.smithi170.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40722.log.gz 2024-03-29T11:23:08.424 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.34749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39027.log 2024-03-29T11:23:08.424 INFO:teuthology.orchestra.run.smithi170.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41505.log.gz 2024-03-29T11:23:08.425 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39729.log 2024-03-29T11:23:08.425 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34749.log.gz 2024-03-29T11:23:08.425 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39027.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:23:08.425 INFO:teuthology.orchestra.run.smithi170.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40918.log.gz 2024-03-29T11:23:08.425 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39729.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39027.log.gz 2024-03-29T11:23:08.425 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39105.log 2024-03-29T11:23:08.425 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph.audit.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39729.log.gz 2024-03-29T11:23:08.425 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41544.log 2024-03-29T11:23:08.425 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40800.log 2024-03-29T11:23:08.425 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38988.log 2024-03-29T11:23:08.425 INFO:teuthology.orchestra.run.smithi170.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39105.log.gz 2024-03-29T11:23:08.425 INFO:teuthology.orchestra.run.smithi170.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:23:08.425 INFO:teuthology.orchestra.run.smithi170.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41544.log.gz 2024-03-29T11:23:08.425 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41739.log 2024-03-29T11:23:08.425 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.38988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41583.log 2024-03-29T11:23:08.425 INFO:teuthology.orchestra.run.smithi170.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40800.log.gz 2024-03-29T11:23:08.426 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39573.log 2024-03-29T11:23:08.426 INFO:teuthology.orchestra.run.smithi170.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38988.log.gz 2024-03-29T11:23:08.426 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41583.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41739.log.gz 2024-03-29T11:23:08.426 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:23:08.426 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40119.log 2024-03-29T11:23:08.426 INFO:teuthology.orchestra.run.smithi170.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41583.log.gz 2024-03-29T11:23:08.426 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39807.log 2024-03-29T11:23:08.426 INFO:teuthology.orchestra.run.smithi170.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39573.log.gz 2024-03-29T11:23:08.426 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39066.log 2024-03-29T11:23:08.426 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40080.log 2024-03-29T11:23:08.426 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40119.log.gz 2024-03-29T11:23:08.426 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39066.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39807.log.gz 2024-03-29T11:23:08.426 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39651.log 2024-03-29T11:23:08.426 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41271.log 2024-03-29T11:23:08.427 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40080.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39066.log.gz 2024-03-29T11:23:08.427 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38949.log 2024-03-29T11:23:08.427 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41271.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40080.log.gz 2024-03-29T11:23:08.427 INFO:teuthology.orchestra.run.smithi170.stderr:gzip 85.4% -- replaced with /var/log/ceph/ceph-client.admin.39651.log.gz -5 2024-03-29T11:23:08.427 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38715.log 2024-03-29T11:23:08.427 INFO:teuthology.orchestra.run.smithi170.stderr: 86.2%/var/log/ceph/ceph-client.admin.38949.log: -- replaced with /var/log/ceph/ceph-client.admin.41271.log.gz 2024-03-29T11:23:08.427 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35040.log 2024-03-29T11:23:08.427 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.38715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39885.log 2024-03-29T11:23:08.427 INFO:teuthology.orchestra.run.smithi170.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38949.log.gz 2024-03-29T11:23:08.427 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.35040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40488.log 2024-03-29T11:23:08.427 INFO:teuthology.orchestra.run.smithi170.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38715.log.gz 2024-03-29T11:23:08.427 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40644.log 2024-03-29T11:23:08.427 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35040.log.gz 2024-03-29T11:23:08.427 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40251.log 2024-03-29T11:23:08.428 INFO:teuthology.orchestra.run.smithi170.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39885.log.gz 2024-03-29T11:23:08.428 INFO:teuthology.orchestra.run.smithi170.stderr:gzip/var/log/ceph/ceph-client.admin.40644.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41193.log 2024-03-29T11:23:08.428 INFO:teuthology.orchestra.run.smithi170.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40488.log.gz 2024-03-29T11:23:08.428 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39378.log 2024-03-29T11:23:08.428 INFO:teuthology.orchestra.run.smithi170.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40644.log.gz 2024-03-29T11:23:08.428 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39183.log 2024-03-29T11:23:08.428 INFO:teuthology.orchestra.run.smithi170.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40251.log.gz 2024-03-29T11:23:08.428 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35143.log 2024-03-29T11:23:08.428 INFO:teuthology.orchestra.run.smithi170.stderr: 87.1%/var/log/ceph/ceph-client.admin.39183.log: -- replaced with /var/log/ceph/ceph-client.admin.41193.log.gz 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39378.log.gz 2024-03-29T11:23:08.428 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-29T11:23:08.428 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:23:08.428 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41113.log 2024-03-29T11:23:08.428 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.35143.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39183.log.gz 2024-03-29T11:23:08.428 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41035.log 2024-03-29T11:23:08.429 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35143.log.gz 2024-03-29T11:23:08.429 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41427.log 2024-03-29T11:23:08.429 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41035.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41113.log.gz 2024-03-29T11:23:08.429 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41856.log 2024-03-29T11:23:08.429 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38871.log 2024-03-29T11:23:08.429 INFO:teuthology.orchestra.run.smithi170.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41035.log.gz 2024-03-29T11:23:08.429 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41310.log 2024-03-29T11:23:08.429 INFO:teuthology.orchestra.run.smithi170.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41427.log.gz 2024-03-29T11:23:08.429 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.38871.log: 86.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.41856.log.gz 2024-03-29T11:23:08.429 INFO:teuthology.orchestra.run.smithi170.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39300.log 2024-03-29T11:23:08.429 INFO:teuthology.orchestra.run.smithi170.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38871.log.gz 2024-03-29T11:23:08.429 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40879.log 2024-03-29T11:23:08.429 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:23:08.429 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39300.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41310.log.gz 2024-03-29T11:23:08.429 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38635.log 2024-03-29T11:23:08.429 INFO:teuthology.orchestra.run.smithi170.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39300.log.gz 2024-03-29T11:23:08.430 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-mgr.y.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40879.log.gz 2024-03-29T11:23:08.430 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40527.log 2024-03-29T11:23:08.430 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39846.log 2024-03-29T11:23:08.430 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.38635.log: /var/log/ceph/ceph-client.admin.40527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40605.log 2024-03-29T11:23:08.430 INFO:teuthology.orchestra.run.smithi170.stderr: 86.6%/var/log/ceph/ceph-client.admin.39846.log: -- replaced with /var/log/ceph/ceph-client.admin.38635.log.gz 2024-03-29T11:23:08.430 INFO:teuthology.orchestra.run.smithi170.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40527.log.gz 2024-03-29T11:23:08.430 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41074.log 2024-03-29T11:23:08.430 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40605.log: gzip -5 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39846.log.gz 2024-03-29T11:23:08.430 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39222.log 2024-03-29T11:23:08.430 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41074.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40605.log.gz 2024-03-29T11:23:08.430 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40566.log 2024-03-29T11:23:08.430 INFO:teuthology.orchestra.run.smithi170.stderr:gzip/var/log/ceph/ceph-client.admin.39222.log: -5 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41074.log.gz 2024-03-29T11:23:08.430 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:23:08.430 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40566.log: gzip -5 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39222.log.gz 2024-03-29T11:23:08.431 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39924.log 2024-03-29T11:23:08.431 INFO:teuthology.orchestra.run.smithi170.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40566.log.gz 2024-03-29T11:23:08.431 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40371.log 2024-03-29T11:23:08.431 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40683.log 2024-03-29T11:23:08.431 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40371.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39924.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38910.log 2024-03-29T11:23:08.431 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-29T11:23:08.431 INFO:teuthology.orchestra.run.smithi170.stderr: 86.3%gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:23:08.431 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40371.log.gz 2024-03-29T11:23:08.431 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40683.log: /var/log/ceph/ceph-client.admin.38910.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40683.log.gz 2024-03-29T11:23:08.431 INFO:teuthology.orchestra.run.smithi170.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38910.log.gz 2024-03-29T11:23:08.431 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38440.log 2024-03-29T11:23:08.431 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39456.log 2024-03-29T11:23:08.431 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.38440.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:23:08.431 INFO:teuthology.orchestra.run.smithi170.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38440.log.gz 2024-03-29T11:23:08.432 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.39456.log: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:23:08.432 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39456.log.gz 2024-03-29T11:23:08.432 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38596.log 2024-03-29T11:23:08.432 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40291.log 2024-03-29T11:23:08.432 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41152.log 2024-03-29T11:23:08.432 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40291.log: /var/log/ceph/ceph-client.admin.38596.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38596.log.gz 2024-03-29T11:23:08.432 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40291.log.gz 2024-03-29T11:23:08.432 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41349.log 2024-03-29T11:23:08.432 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41817.log 2024-03-29T11:23:08.432 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41349.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41152.log.gz 2024-03-29T11:23:08.432 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40761.log 2024-03-29T11:23:08.432 INFO:teuthology.orchestra.run.smithi170.stderr: 86.2%/var/log/ceph/ceph-client.admin.41817.log: -- replaced with /var/log/ceph/ceph-client.admin.41349.log.gz 2024-03-29T11:23:08.432 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- 85.8% /var/log/ceph/ceph-client.admin.40041.log 2024-03-29T11:23:08.432 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41817.log.gz 2024-03-29T11:23:08.432 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40957.log 2024-03-29T11:23:08.432 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40761.log: /var/log/ceph/ceph-client.admin.40041.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40761.log.gz 2024-03-29T11:23:08.433 INFO:teuthology.orchestra.run.smithi170.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40041.log.gz 2024-03-29T11:23:08.433 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41388.log 2024-03-29T11:23:08.433 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40410.log 2024-03-29T11:23:08.433 INFO:teuthology.orchestra.run.smithi170.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40957.log.gz 2024-03-29T11:23:08.433 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.41388.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.41388.log.gz 2024-03-29T11:23:08.433 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38518.log 2024-03-29T11:23:08.433 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.40410.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:23:08.433 INFO:teuthology.orchestra.run.smithi170.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40410.log.gz 2024-03-29T11:23:08.433 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.38518.log: /var/log/ceph/ceph-mon.c.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38518.log.gz 2024-03-29T11:23:08.435 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:23:08.435 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34473.log 2024-03-29T11:23:08.436 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:23:08.436 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34393.log 2024-03-29T11:23:08.436 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34353.log 2024-03-29T11:23:08.436 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.34473.log: /var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.34393.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:23:08.436 INFO:teuthology.orchestra.run.smithi184.stderr: 87.8% 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz -- replaced with /var/log/ceph/ceph-client.admin.34473.log.gz 2024-03-29T11:23:08.436 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:23:08.436 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.34353.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:23:08.436 INFO:teuthology.orchestra.run.smithi184.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34393.log.gz 2024-03-29T11:23:08.436 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:23:08.436 INFO:teuthology.orchestra.run.smithi184.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34353.log.gz 2024-03-29T11:23:08.436 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34433.log 2024-03-29T11:23:08.436 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34233.log 2024-03-29T11:23:08.436 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.34433.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:23:08.436 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.34233.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34433.log.gz 2024-03-29T11:23:08.436 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:23:08.436 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-mgr.x.log: gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34233.log.gz 2024-03-29T11:23:08.437 INFO:teuthology.orchestra.run.smithi184.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34313.log 2024-03-29T11:23:08.437 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:23:08.437 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.34313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34513.log 2024-03-29T11:23:08.437 INFO:teuthology.orchestra.run.smithi184.stderr: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:23:08.437 INFO:teuthology.orchestra.run.smithi184.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34313.log.gz 2024-03-29T11:23:08.437 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34273.log 2024-03-29T11:23:08.437 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.34513.log: /var/log/ceph/ceph-client.admin.34273.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34513.log.gz 2024-03-29T11:23:08.437 INFO:teuthology.orchestra.run.smithi184.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34273.log.gz 2024-03-29T11:23:08.437 INFO:teuthology.orchestra.run.smithi184.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:23:08.462 INFO:teuthology.orchestra.run.smithi170.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:23:19.889 INFO:teuthology.orchestra.run.smithi170.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:23:19.993 INFO:teuthology.orchestra.run.smithi170.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:23:20.061 INFO:teuthology.orchestra.run.smithi184.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:23:20.268 INFO:teuthology.orchestra.run.smithi184.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:23:20.284 INFO:teuthology.orchestra.run.smithi184.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:23:23.945 INFO:teuthology.orchestra.run.smithi170.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:23:24.014 INFO:teuthology.orchestra.run.smithi170.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:23:24.411 INFO:teuthology.orchestra.run.smithi184.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:23:58.556 INFO:teuthology.orchestra.run.smithi170.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:24:01.338 INFO:teuthology.orchestra.run.smithi170.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:24:01.340 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-29T11:24:01.340 INFO:teuthology.orchestra.run.smithi170.stderr:real 0m52.981s 2024-03-29T11:24:01.340 INFO:teuthology.orchestra.run.smithi170.stderr:user 2m31.681s 2024-03-29T11:24:01.340 INFO:teuthology.orchestra.run.smithi170.stderr:sys 0m5.511s 2024-03-29T11:24:18.620 INFO:teuthology.orchestra.run.smithi184.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:24:18.621 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-29T11:24:18.622 INFO:teuthology.orchestra.run.smithi184.stderr:real 1m10.221s 2024-03-29T11:24:18.622 INFO:teuthology.orchestra.run.smithi184.stderr:user 1m57.136s 2024-03-29T11:24:18.622 INFO:teuthology.orchestra.run.smithi184.stderr:sys 0m4.269s 2024-03-29T11:24:18.622 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:24:18.622 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622678/remote/smithi170/log 2024-03-29T11:24:18.624 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:24:41.027 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622678/remote/smithi184/log 2024-03-29T11:24:41.028 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:24:59.452 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:24:59.660 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b13531e8a44b4d60a91d653b3cd54ba4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:24:59.663 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:24:59.674 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:24:59.687 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:24:59.687 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:24:59.701 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:24:59.779 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:24:59.788 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:24:59.789 DEBUG:teuthology.orchestra.run.smithi170:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:24:59.791 DEBUG:teuthology.orchestra.run.smithi184:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:24:59.806 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:24:59.807 INFO:teuthology.orchestra.run.smithi184.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:24:59.811 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:24:59.812 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-03-29T11:24:59.812 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 54 +433us[ +443us] +/- 84ms 2024-03-29T11:24:59.812 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 54 -143us[ -133us] +/- 79ms 2024-03-29T11:24:59.812 INFO:teuthology.orchestra.run.smithi170.stdout:^* hv03.front.sepia.ceph.com 3 6 377 52 -255us[ -244us] +/- 79ms 2024-03-29T11:24:59.813 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:24:59.814 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:24:59.815 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2024-03-29T11:24:59.815 INFO:teuthology.orchestra.run.smithi184.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 55 +436us[ +441us] +/- 84ms 2024-03-29T11:24:59.815 INFO:teuthology.orchestra.run.smithi184.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 54 -137us[ -132us] +/- 79ms 2024-03-29T11:24:59.815 INFO:teuthology.orchestra.run.smithi184.stdout:^* hv03.front.sepia.ceph.com 3 6 377 52 -249us[ -243us] +/- 79ms 2024-03-29T11:24:59.816 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:24:59.817 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:24:59.826 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:24:59.827 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:24:59.839 DEBUG:teuthology.orchestra.run.smithi170:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:24:59.924 DEBUG:teuthology.orchestra.run.smithi184:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:24:59.996 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:25:00.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:25:00.028 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:25:00.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:25:00.061 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:25:00.071 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:25:00.082 INFO:teuthology.task.internal:Duration was 1351.556319 seconds 2024-03-29T11:25:00.083 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:25:00.095 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:25:00.095 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:25:00.098 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:25:00.140 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:25:00.146 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:25:00.458 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:25:00.459 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi170.front.sepia.ceph.com 2024-03-29T11:25:00.459 DEBUG:teuthology.orchestra.run.smithi170:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:25:00.479 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi184.front.sepia.ceph.com 2024-03-29T11:25:00.479 DEBUG:teuthology.orchestra.run.smithi184:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:25:00.501 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:25:00.501 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:25:00.521 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:25:00.570 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:25:00.571 DEBUG:teuthology.orchestra.run.smithi170:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:25:00.589 DEBUG:teuthology.orchestra.run.smithi184:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:25:00.707 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:25:00.716 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:25:00.717 DEBUG:teuthology.orchestra.run.smithi170:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:25:00.744 DEBUG:teuthology.orchestra.run.smithi184:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:25:00.771 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:25:00.780 DEBUG:teuthology.orchestra.run.smithi170:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:25:00.787 DEBUG:teuthology.orchestra.run.smithi184:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:25:00.809 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = core 2024-03-29T11:25:00.836 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = core 2024-03-29T11:25:00.858 DEBUG:teuthology.orchestra.run.smithi170:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:25:00.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:25:00.892 DEBUG:teuthology.orchestra.run.smithi184:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:25:00.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:25:00.912 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:25:00.923 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:25:00.925 DEBUG:teuthology.misc:Transferring archived files from smithi170:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622678/remote/smithi170 2024-03-29T11:25:00.925 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:25:00.968 DEBUG:teuthology.misc:Transferring archived files from smithi184:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622678/remote/smithi184 2024-03-29T11:25:00.968 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:25:01.004 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:25:01.005 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:25:01.007 DEBUG:teuthology.orchestra.run.smithi184:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:25:01.054 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:25:01.066 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:25:01.066 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:25:01.079 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:25:01.079 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:25:01.082 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:25:01.094 INFO:teuthology.orchestra.run.smithi170.stdout: 266042 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:25 /home/ubuntu/cephtest 2024-03-29T11:25:01.109 INFO:teuthology.orchestra.run.smithi184.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:25 /home/ubuntu/cephtest 2024-03-29T11:25:01.110 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:25:01.121 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:25:01.146 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:25:01.173 DEBUG:teuthology.nuke:shortname: smithi170 2024-03-29T11:25:01.174 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:25:01.197 DEBUG:teuthology.nuke:shortname: smithi184 2024-03-29T11:25:01.198 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:25:01.201 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622678', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:54:10.697203', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK8KF8LxGDTr84GrjnvlzOw6qKh2eoa6jz9fKWQDjuMowPMVTuuGN5hLqxNs4JjbYvSy47O9KcB53fNM3ZmiQp8='} 2024-03-29T11:25:01.224 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622678', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:54:10.695910', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBChIWo4sRluXRqRfde3HW9mpN2k3+EawSGeT0X7df0O5eilEPv32w+Elg4a1z+ulVm2/6dGkwu0juAUsVblJt10='} 2024-03-29T11:25:01.253 INFO:teuthology.orchestra.console.smithi170:Power off 2024-03-29T11:25:01.253 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:25:01.297 INFO:teuthology.orchestra.console.smithi184:Power off 2024-03-29T11:25:01.298 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:25:01.308 DEBUG:teuthology.orchestra.console.smithi170:power off output: Chassis Power Control: Down/Off 2024-03-29T11:25:01.309 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:25:01.321 DEBUG:teuthology.orchestra.console.smithi184:power off output: Chassis Power Control: Down/Off 2024-03-29T11:25:01.322 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:25:01.333 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2024-03-29T11:25:01.346 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-03-29T11:25:05.334 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:25:05.346 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:25:05.446 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2024-03-29T11:25:05.457 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-03-29T11:25:09.450 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:25:09.460 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:25:09.561 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off 2024-03-29T11:25:09.570 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-03-29T11:25:09.663 INFO:teuthology.orchestra.console.smithi170:Power off completed 2024-03-29T11:25:13.574 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:25:13.686 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is off 2024-03-29T11:25:13.787 INFO:teuthology.orchestra.console.smithi184:Power off completed 2024-03-29T11:25:13.891 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1351.5563192367554 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b13531e8a44b4d60a91d653b3cd54ba4 status: fail success: false 2024-03-29T11:25:13.892 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:25:13.982 INFO:teuthology.run:FAIL